Add support for Apple M1 (amd64) architecture

[alancleary]

The docker compose files will not work in their current form on Apple computers that have an M1/M2 processor (amd64 architecture). This is because not all of the images specified in the compose files support the arm64 architecture. Specifically, we currently aren't building containers that support arm64 and the official RediSearch image doesn't support arm64.

Since we don't have control over what architectures the RediSearch image supports, we can get around this by explicitly specifying platform: linux/amd64 for the RediSearch image and updating the readme to inform users with M1/M2 processors that they need to install Rosetta 2 so they can emulate the amd64 architecture. We should lean on this solution for our GCV images too since the most common use case is deploying the microservices on a Linux server.

@adf-ncgr can you confirm that this solution worked on your Apple computer with an M1/M2 processor?

[adf-ncgr]

@alancleary no, I don't think it does work. I added the platform directive to the redisearch service in the compose.yml file of a tagged release:

  redisearch:
    image: redislabs/redisearch:2.4.14
    platform: linux/amd64

upon startup with a dump.rdb file in place I get a long error in logs (elided here):

=== REDIS BUG REPORT START: Cut & paste starting from here === 1:M 28 Sep 2022 19:19:53.034 # Redis 6.2.7 crashed by signal: 11, si_code: 1 1:M 28 Sep 2022 19:19:53.034 # Accessing address: (nil) 1:M 28 Sep 2022 19:19:53.034 # Crashed running the instruction at: 0x400006af2d .... qemu: uncaught target signal 11 (Segmentation fault) - core dumped

this seems similar to what is described in: https://github.com/RediSearch/RediSearch/issues/2367

the docker-compose docs have this to say about emulation:

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

I am vaguely thinking that I had had some success in running GCV on the M1 previously, but it certainly could be that I was just running the client and pointing the config to services hosted elsewhere. Although it doesn't look like the redisearch maintainers are responsive to the issue, I don't think you probably ought to be attempting any heroics to get this working, unless you are seriously concerned that there are hordes (>0) of potential GCV developers who are only being deterred by this issue...

[alancleary]

That's too bad; I didn't realize the amd64 emulation was unreliable.

Before opening this issue I found an issue in the RediSearch repo that refers to a RediSearch image on Docker Hub that supports arm64. While the person who recommended the Docker image on GitHub is a contributor to RediSearch and the image seems to be kept up to date on Docker Hub, I'm hesitant to use it because I can't determine who redisfab is or how they're affiliated with the Redis project (if at all), so who knows what's actually in that Docker image!

If we do decide to use that RediSearch image then we'll need to decide how to proceed with the GCV Docker images, i.e. confirm that they work with Rosetta or that we can compile them to support amd64.

[adf-ncgr]

I guess if avatars are anything to go by, then https://hub.docker.com/u/redisfab and https://github.com/rafie are probably one and the same, but I'm certainly not going to tell you to trust the image on the basis of that level of authentication. my grasp of the issues here is tenuous at best. I suspect that @nathanweeks will be the best person to provide guidance on whether and how to move forward with multi-arch support...

[alancleary]

After doing some reading on multi architecture builds I think building an image for each GCV service that supports multiple architectures is the way to go. Specifically, this would be done with docker buildx, which wouldn't require any additional configuration files or modifications to the services' Dockerfile files: https://docs.docker.com/build/building/multi-platform/.

I'll need to play around with buildx to better understand how it works. If it's truly the path forward then we'll need to update the GitHub action we're using to automate builds since the action we're currently using uses docker build instead of buildx. As a silver lining, the action I found that uses buildx instead of build was actually created by docker, whereas the action we're currently using was not created by docker. So if we go this route we'll be using official tooling :tada:

[nathanweeks]

Before opening this issue I found an issue in the RediSearch repo that refers to a RediSearch image on Docker Hub that supports arm64. While the person who recommended the Docker image on GitHub is a contributor to RediSearch and the image seems to be kept up to date on Docker Hub, I'm hesitant to use it because I can't determine who redisfab is or how they're affiliated with the Redis project (if at all), so who knows what's actually in that Docker image!

The README in the RediSearch/RediSearch repo now suggests trying RediSearch using the Redis Stack Docker Image (redis/redis-stack-server), which appears to have linux/arm64 images. That might be worth a try?

[alancleary]

Thanks @nathanweeks! That looks really promising. As a bonus, the image comes loaded with the other official Redis modules, not just RediSearch. This may be useful as we push GCV forward.

[alancleary]

Commit 4a59c8ebcf120d5856221574dab3218365fe8f58 replaced the redis container with the redis-stack-server container @nathanweeks mentioned. This brings us one step closer to supporting amd64.

[adf-ncgr]

FWIW, this still seems to be an issue for me on M1 (not that I'm upping the priority, just venting after realizing that it was the explanation for bad behaviors I was seeing when trying to test what I was doing for /legumeinfo/gcv-docker-compose/#6)