search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.3k stars 14.09k forks source link

Make "multi-platform" images build/workflow works back using ARM and AMD instances #41935

Open potiuk opened 1 week ago

potiuk commented 1 week ago

Currently the workflow to build multi-platform images are disabled (see UPDATE: below - they are enabled back but using slow emulation). The problem is that we need to have two platforms connected with SSH and it stopped working when "self-hosted" runners stopped working aftet patching GitHub runners done by @ashb stopped working.

The workflow that used to work is: https://github.com/apache/airflow/blob/main/.github/workflows/release_dockerhub_image.yml

With the ARC ARM runners we should be able to bring it back but differently.

The current multi-platform image building (described in https://github.com/apache/airflow/blob/main/dev/MANUALLY_BUILDING_IMAGES.md uses buildkit capability to build multi-platform images using multiple docker workers. By running the commands with --dry-run you could see the exact docker build commands that build the images.

However, it's possible to build airflow images using separate, non-connected runners (one for AMD and one for ARM) - by pushing images to docker and building a common manifest. This is described in https://docs.docker.com/build/ci/github-actions/multi-platform/ and with ARC runners, we should be able to do it.

potiuk commented 1 week ago

UPDATE: as of https://github.com/apache/airflow/pull/41959, the workflows are working back with emulation support - but they are REALLY slow > 1hr comparing to 15 minutes when ARM hardware is used.