Docker preview version for apple silicon is fail to run 'apache/airflow' image

kevin1172 commented 3 years ago

Issue:

I tried to run the 'apache/airflow' image (https://hub.docker.com/r/apache/airflow) with docker-compose. The container is able to start, but after running some airflow commends, the container is terminated with an error 'qemu: uncaught target signal 4 (Illegal instruction) - core dumped'

Actual behavior

Following is my docker-compose.yml file

version: '3.8'
services:
    webserver:
        image: apache/airflow
        command: bash -c
            "airflow db init 
            && airflow users create -u admin -f Peter -l Parker -r Admin -e spiderman@superhero.org -p ****
            && airflow webserver"
        volumes:
            - ./dags:/opt/airflow/dags
        environment:
            - AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql+mysqldb://airflow:airflow@192.168.64.1:3306/airflow
            - AIRFLOW__CORE__EXECUTOR=LocalExecutor
        ports:
            - "8080:8080"

After I run "docker-compose up", I get:

Recreating docker-airflow_webserver_1 ... done
Attaching to docker-airflow_webserver_1
webserver_1  | DB_BACKEND=mysql+mysqldb
webserver_1  | DB_HOST=192.168.64.1
webserver_1  | DB_PORT=3306
webserver_1  | 
webserver_1  | DB: mysql+mysqldb://airflow:***@192.168.64.1:3306/airflow
webserver_1  | [2021-01-08 04:28:45,451] {db.py:678} INFO - Creating tables
webserver_1  | INFO  [alembic.runtime.migration] Context impl MySQLImpl.
webserver_1  | INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
webserver_1  | qemu: uncaught target signal 4 (Illegal instruction) - core dumped
webserver_1  | /bin/bash: line 1:    21 Illegal instruction     airflow db init
docker-airflow_webserver_1 exited with code 132

Information

macOS Version: Big Sur
Docker for Mac: Apple silicon preview version released on Dec 16

Do I have to wait for the production version of Docker Desktop for Mac? How can I fix it? Thanks!

stephen-turner commented 3 years ago

This is a qemu bug, which is the upstream component we use for running Intel (amd64) containers on M1 (arm64) chips, and is unfortunately not something we control. In general we recommend running arm64 containers on M1 chips because (even ignoring any crashes) they will always be faster and use less memory.

Please encourage the author of this container to supply an arm64 or multi-arch image, not just an Intel one. Now that M1 is a mainstream platform, we think that most container authors will be keen to do this.

ashb commented 3 years ago

:wave: Airflow Dev here.

Any pointers as to how we do that, especially without access to an M1 machine?

stephen-turner commented 3 years ago

Hi @ashb, thanks for asking. You should be able to use buildx to build for another architecture, or even build a multi-architecture image that targets several architectures at once. Have a look at our documentation https://docs.docker.com/docker-for-mac/multi-arch/ for that (or there are loads of blog posts if you search for "docker multi-arch build").

Hopefully that will work even though it uses qemu, which crashed in the original bug report. If not, you would need to build on an ARM machine, but not necessarily an M1: AWS has ARM machines, for example.

docker-robott commented 3 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked

docker / for-mac