Make pulling images on M1 / ARM silicon more user friendly

[RobbeSneyders]

Currently users who try to pull images on their M1 silicon (eg. using the local runner) will run into the following issue:

no matching manifest for linux/arm64/v8 in the manifest list entries

They can solve this by setting the DOCKER_DEFAULT_PLATFORM environment variable.

export DOCKER_DEFAULT_PLATFORM=linux/amd64

This will instruct docker to pull images built for the linux/amd64 platform instead. From the docker docs:

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. This issue should become less common over time, as more and more images are rebuilt supporting multiple architectures.

I see different actions we can take with different levels of investment and return:

• Document the current workaround clearly
• Add the platform flag / environment variable in the docker compose file when compiling the pipeline
• Build arm64 or multi-arch images. We should investigate what is needed for this and how it will affect our build times, which are already quite long building for a single architecture.

  
  ### Tasks

• [ ] https://github.com/ml6team/fondant/issues/690

[GeorgesLorre]

Building multi-arch images has nice benefits:

• easiest to use (no setting of env vars)
• best performance (no emulation)
• can be expanded for specialized architecture (TPU, AWS Graviton)

Downsides:

• extra build time (cache seems to be unique per architecture) -> does this fit in github actions
• extra problems that arise because of differences between architectures

[GeorgesLorre]

If it is only for local development (local runner on Mac's M1 chips) then I would just alter the docker-compose file to include the platform.

https://github.com/compose-spec/compose-spec/blob/master/spec.md#platform

[RobbeSneyders]

The major clouds also offer ARM machines, especially Amazon it seems. Not sure how popular those are, but those would benefit from multi-arch images as well.

[RobbeSneyders]

Since this might have an impact on image size as well, let's take https://github.com/ml6team/fondant/issues/573 into account and maybe tackle them together.

[GeorgesLorre]

As a temporary fix we will add the:

export DOCKER_DEFAULT_PLATFORM=linux/amd64

on the fondant side. Until we are ready to build ARM images (with reasonable size)