internetstandards / Internet.nl

Internet standards compliance test suite
https://internet.nl
171 stars 35 forks source link

Improve support for Apple ARM/Rosetta #1301

Open mxsasha opened 5 months ago

mxsasha commented 5 months ago

The current Docker image uses x86_64 as the colima architecture which works on Apple ARM macs but is unusably slow. It doesn't make optimal use of Rosetta. It is possible to run colima as aarch64, with arch: aarch64, vmType: vz and rosetta: true. THe containers need to be amd64 still, as our nassl does not compile at all on aarch64, and newer nassl also seems tricky. I may have run into some other issues as well. To avoid this, I set all containers to amd64 using the patch below, and that's fine, as they'll use Rosetta with these colima settings, meaning they'll have good performance. However, presumably from the switch from qemu to vz, the containers lose IPv6 connectivity with this. Note the colima VM still has v6, i.e. colima ssh ping6 2600:: works, but docker exec -ti internetnl-develop-app-1 ping6 2600:: does not. The integration test fully succeeds and is fast.

It would be great to have both Rosetta and IPv6 support on macs, but do not see a clear way. Perhaps @aequitas has more input?

Some references:

diff --git a/docker/Dockerfile b/docker/Dockerfile
index 549d83a..6a4a55b 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -10,7 +10,7 @@ ARG PYTHON_VERSION=3.9
 # hashes/log output prefix build command with: `DOCKER_BUILDKIT=0`

 # generic stage used by all build stages
-FROM debian:bullseye-20230502-slim as build-deps
+FROM --platform=linux/amd64 debian:bullseye-20230502-slim as build-deps
 ARG PYTHON_VERSION

 RUN apt update && \
@@ -75,7 +75,7 @@ COPY requirements-dev.txt /src/requirements-dev.txt
 RUN pip3 install -r requirements-dev.txt

 # build unbound target
-FROM debian:bullseye-20230502-slim as unbound
+FROM --platform=linux/amd64 debian:bullseye-20230502-slim as unbound

 COPY --from=build-unbound /opt/unbound /opt/unbound

@@ -123,7 +123,7 @@ USER unbound
 ENTRYPOINT ["/entrypoint.sh"]

 # build main application image target
-FROM debian:bullseye-20230502-slim as build-app
+FROM --platform=linux/amd64 debian:bullseye-20230502-slim as build-app
 ARG PYTHON_VERSION

 RUN apt update && \
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index ae7945b..cb8b75b 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -65,6 +65,7 @@ services:
       retries: 10

   app:
+    platform: "linux/amd64"
     image: ${DOCKER_IMAGE_APP:-${DOCKER_REGISTRY:-ghcr.io/internetstandards}/internet.nl:${RELEASE:-latest}}
     build:
       context: ..
@@ -170,6 +171,7 @@ services:
       - manual-hof:/app/manual-hall-of-fame/

   db-migrate:
+    platform: "linux/amd64"
     image: ${DOCKER_IMAGE_APP:-${DOCKER_REGISTRY:-ghcr.io/internetstandards}/internet.nl:${RELEASE:-latest}}
     build:
       context: ..
@@ -208,6 +210,7 @@ services:
       - WORKER_CONCURRENCY

   worker: &worker
+    platform: "linux/amd64"
     image: ${DOCKER_IMAGE_APP:-${DOCKER_REGISTRY:-ghcr.io/internetstandards}/internet.nl:${RELEASE:-latest}}
     build:
       context: ..
@@ -332,6 +335,7 @@ services:
     hostname: worker-slow

   beat:
+    platform: "linux/amd64"
     image: ${DOCKER_IMAGE_APP:-${DOCKER_REGISTRY:-ghcr.io/internetstandards}/internet.nl:${RELEASE:-latest}}
     build:
       context: ..