Provide arm64 docker images

wagnerand commented 1 year ago

On Macs using the Apple Silicon arm-based architecture, Docker raises warnings for several containers about performance impact.

I can confirm that running tests on a M2 Pro are very slow. Can we provide arm64 images for Apple Silicon users?

┆Issue is synchronized with this Jira Task

diox commented 1 year ago

As I commented in the pull request, the problem is that MySQL doesn't provide arm64 deb packages, and that using the Debian built-in mysql packages actually pulls mariadb ones, which have slight incompatibilities that we ran into in the past.

We could try using the non-deb binaries, or build the deb packages ourselves...

An alternative to try though, is on the host, enabling the (beta) feature Rosetta for x86/amd64 emulation in docker. It should enable us to keep AMD64 images while still getting the perf improvements you seek.

vroooooooum-docker-rosetta

(Not pictured on this screenshot: step 4, which is to click Apply & restart :) )

wagnerand commented 1 year ago

With that, docker compose up -d fails with:

 ⠏ memcached The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested      
 ⠏ addons-frontend The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ⠏ web The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested            
 ⠏ autograph The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested      
 ⠇ worker The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Would you accept a patch that explicitly sets platform: linux/amd64 for those images in docker-compose.yaml?

diox commented 1 year ago

Would just setting DOCKER_DEFAULT_PLATFORM on the host instead work ? And more importantly, does that solve your perf issues ?

wagnerand commented 1 year ago

I'm not sure this is desirable for several reasons: It would pull amd64 images even if arm64 images are available. Also, every contributor on an Apple Silicon device would have to know and set this flag on their machine. We can document it on readthedocs, but it still leaves a visibility problem.

I can run the full test suite some time and see how long it takes.

wagnerand commented 1 year ago

Using platform.patch I ran the full test suite:

Docker images:

Without rosetta: 27 failed, 6576 passed, 2 xfailed, 1644 warnings in 5927.57s (1:38:47) With rosetta: 28 failed, 6575 passed, 2 xfailed, 1644 warnings in 1399.62s (0:23:19)

(Failures difference: src/olympia/search/tests/test_utils.py::TestUnindexObjects::test_unindex_objects - AssertionError: ['3', '5', '9', '11', '13', '16', ...])

When using export DOCKER_DEFAULT_PLATFORM=linux/amd64 instead of the patch above, all containers are amd64, so performance will be either equal or worse.

When using neither the patch nor the platform override environment variable, pulling the docker images fails.

diox commented 1 year ago

Let's go with your patch adding platform: to the relevant services in docker-compose.yml. Could you submit a PR for that ?

KevinMind commented 4 months ago

Old Jira Ticket: https://mozilla-hub.atlassian.net/browse/ADDSRV-256

mozilla / addons

Provide arm64 docker images #1919