search

astronomer / ap-airflow

Astronomer Core Docker Images
https://astronomer.io
Other
106 stars 29 forks source link

Unable to build latest Docker images #471

Open sfc-gh-cbaechtold opened 2 years ago

sfc-gh-cbaechtold commented 2 years ago

I am no longer able to build the latest Airflow image on Docker for Desktop version 4.6.1 (76265), running Mac OS Monterey v12.3.1.

I previously had been able to successfully build the 2.2.5 release, but that now fails. I also tried building the 2.3.0 release, and that also failed. Both due to pip being unable to find a version that satisfied requirements for astronomer-certified[.....]==2.2.5-3-* (and 2.3.0 similarly). Full stack trace:

> DOCKER_BUILDKIT=0 docker build --build-arg PYTHON_MAJOR_MINOR_VERSION=3.8 -t ap-airflow:py38 https://github.com/astronomer/ap-airflow.git#master:2.3.0/bullseye/
Sending build context to Docker daemon  30.21kB
Step 1/67 : ARG APT_DEPS_IMAGE="airflow-apt-deps"
Step 2/67 : ARG PYTHON_MAJOR_MINOR_VERSION="3.9"
Step 3/67 : ARG PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-bullseye"
Step 4/67 : FROM ${PYTHON_BASE_IMAGE} as airflow-apt-deps
 ---> 654e18c3f59c
Step 5/67 : LABEL maintainer="Astronomer <humans@astronomer.io>"
 ---> Using cache
 ---> 1c96172c8065
Step 6/67 : ARG ASTRONOMER_USER="astro"
 ---> Using cache
 ---> 204eabfc3b98
Step 7/67 : ARG ASTRONOMER_UID="50000"
 ---> Using cache
 ---> 83b112225176
Step 8/67 : LABEL io.astronomer.docker=true
 ---> Using cache
 ---> e46957e97ba2
Step 9/67 : LABEL io.astronomer.docker.distro="debian"
 ---> Using cache
 ---> 12ae0a3d4812
Step 10/67 : LABEL io.astronomer.docker.module="airflow"
 ---> Using cache
 ---> dc8791d042f7
Step 11/67 : LABEL io.astronomer.docker.component="airflow"
 ---> Using cache
 ---> 50829d950c62
Step 12/67 : LABEL io.astronomer.docker.uid="${ASTRONOMER_UID}"
 ---> Using cache
 ---> b9d8acf44b83
Step 13/67 : ARG ORG="astronomer"
 ---> Using cache
 ---> a999463fbeb3
Step 14/67 : ENV AIRFLOW_HOME="/usr/local/airflow"
 ---> Using cache
 ---> bb61da75bc45
Step 15/67 : ENV PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}${AIRFLOW_HOME}
 ---> Using cache
 ---> b971365c26b6
Step 16/67 : ENV ASTRONOMER_USER=${ASTRONOMER_USER}
 ---> Using cache
 ---> af6e48608745
Step 17/67 : ENV ASTRONOMER_UID=${ASTRONOMER_UID}
 ---> Using cache
 ---> 71b33d71cab5
Step 18/67 : ARG PYTHON_BASE_IMAGE
 ---> Using cache
 ---> 3e5b784bb97b
Step 19/67 : ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
 ---> Using cache
 ---> 6d795160f3c2
Step 20/67 : ARG PYTHON_MAJOR_MINOR_VERSION
 ---> Using cache
 ---> 639c51da0550
Step 21/67 : ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
 ---> Using cache
 ---> d15a535a7564
Step 22/67 : ARG PIP_VERSION="21.2.4"
 ---> Using cache
 ---> 408e6dc75a18
Step 23/67 : ENV PYTHON_PIP_VERSION=${PIP_VERSION}
 ---> Using cache
 ---> c36cc72b62d8
Step 24/67 : ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8     LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
 ---> Using cache
 ---> 40057402fe74
Step 25/67 : ARG DEPENDENCIES_EPOCH_NUMBER="2"
 ---> Using cache
 ---> 676cbb5c9126
Step 26/67 : ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
 ---> Using cache
 ---> 5f6cc1bed465
Step 27/67 : RUN apt-get update     && apt-get install -y --no-install-recommends            apt-utils            curl            libmariadb3            freetds-bin            gosu            libffi7            libkrb5-3            libpq5            libsasl2-2            libsasl2-modules            libssl1.1            locales             netcat            rsync            sasl2-bin            sudo            tini     && apt-get autoremove -yqq --purge     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 9cf0c3ac1a3b
Step 28/67 : RUN pip install --upgrade pip=="${PYTHON_PIP_VERSION}"
 ---> Using cache
 ---> 9461618a6cb2
Step 29/67 : RUN useradd --uid $ASTRONOMER_UID --create-home ${ASTRONOMER_USER}     && groupadd astrogroup --gid 101     && usermod --append --groups astrogroup ${ASTRONOMER_USER}
 ---> Using cache
 ---> a34cf6859437
Step 30/67 : FROM ${APT_DEPS_IMAGE} as devel
 ---> a34cf6859437
Step 31/67 : SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
 ---> Using cache
 ---> eefe33e545d3
Step 32/67 : ENV PIP_NO_CACHE_DIR="true"
 ---> Using cache
 ---> b26b8b1eec8d
Step 33/67 : RUN apt-get update     && curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -     && apt-get install -y --no-install-recommends         build-essential         default-libmysqlclient-dev         libffi-dev         libkrb5-dev         libpq-dev         libsasl2-dev         libssl-dev     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 44d9a1d678b5
Step 34/67 : ARG VERSION="2.3.0-1-*"
 ---> Running in fe07adf2edb0
Removing intermediate container fe07adf2edb0
 ---> ce1b787845ca
Step 35/67 : ARG SUBMODULES="async,azure,amazon,elasticsearch,google,password,cncf.kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
 ---> Running in 466474de123b
Removing intermediate container 466474de123b
 ---> 3d5e855f1f0b
Step 36/67 : ARG AIRFLOW_MODULE="astronomer_certified[${SUBMODULES}]==$VERSION"
 ---> Running in f53f2b4b1a74
Removing intermediate container f53f2b4b1a74
 ---> 48d77d872f25
Step 37/67 : ARG AIRFLOW_VERSION="2.3.0"
 ---> Running in e295064c7e0f
Removing intermediate container e295064c7e0f
 ---> c91edae1de05
Step 38/67 : ARG ASTRONOMER_FAB_SECURITY_MANAGER_VERSION="1.9.1"
 ---> Running in 21bea3cde6ff
Removing intermediate container 21bea3cde6ff
 ---> 5459a53eb263
Step 39/67 : COPY include/pip.conf /etc/pip.conf
 ---> 70ee97bac288
Step 40/67 : COPY include/pip-constraints.txt /usr/local/share/astronomer-pip-constraints.txt
 ---> a0467429a7df
Step 41/67 : COPY build-time-pip-constraints.txt /tmp/build-time-pip-constraints.txt
 ---> 0a82e8991f77
Step 42/67 : RUN pip install "${AIRFLOW_MODULE}" celery flower         --constraint /tmp/build-time-pip-constraints.txt     && pip install "https://github.com/astronomer/astronomer-airflow-scripts/releases/download/v0.0.5/astronomer_airflow_scripts-0.0.5-py3-none-any.whl"     && pip install "astronomer-fab-security-manager==${ASTRONOMER_FAB_SECURITY_MANAGER_VERSION}"
 ---> Running in 7ac63e8d3338
+ pip install 'astronomer_certified[async,azure,amazon,elasticsearch,google,password,cncf.kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==2.3.0-1-*' celery flower --constraint /tmp/build-time-pip-constraints.txt
Looking in indexes: https://pypi.org/simple, https://pip.astronomer.io/simple/
ERROR: Could not find a version that satisfies the requirement astronomer_certified[amazon,async,azure,cncf.kubernetes,elasticsearch,google,mysql,password,postgres,redis,slack,ssh,statsd,virtualenv]==2.3.0-1-* (from versions: 1.10.7.post7, 1.10.7.post8, 1.10.7.post9, 1.10.7.post10, 1.10.7.post11, 1.10.7.post12, 1.10.7.post13, 1.10.7.post14, 1.10.7.post15, 1.10.7.post16, 1.10.7.post17, 1.10.7.post18, 1.10.7.post19, 1.10.10.post1, 1.10.10.post2, 1.10.10.post3, 1.10.10.post4, 1.10.10.post5, 1.10.10.post6, 1.10.10.post7, 1.10.10.post8, 1.10.10.post9, 1.10.10.post10, 1.10.12.post1, 1.10.12.post2, 1.10.12.post3, 1.10.12.post4, 1.10.12.post5, 1.10.12.post6, 1.10.13.post1, 1.10.14.post1, 1.10.14.post2, 1.10.14.post3, 1.10.14.post4, 1.10.14.post5, 1.10.15.post1, 1.10.15.post2, 1.10.15.post3, 1.10.15.post4, 1.10.15.post5.dev3, 1.10.15.post5, 1.10.15.post6.dev1, 1.10.15.post6, 1.10.15.post7.dev1, 1.10.15.post7, 2.0.0.post1, 2.0.0.post2, 2.0.0.post3, 2.0.0.post4, 2.0.0.post5, 2.0.0.post6, 2.0.0.post7, 2.0.0.post8, 2.0.0.post9, 2.0.0.post10, 2.0.1.post1, 2.0.2.post1, 2.0.2.post2, 2.0.2.post3, 2.0.2.post4, 2.0.2.post5, 2.0.2.post6, 2.1.0.post1, 2.1.0.post2, 2.1.0.post3, 2.1.0.post4, 2.1.0.post5, 2.1.0.post6, 2.1.0.post7.dev2, 2.1.0.post7.dev5, 2.1.0.post7, 2.1.1.post1, 2.1.1.post2, 2.1.1.post3, 2.1.1.post4, 2.1.1.post5, 2.1.1.post6.dev2, 2.1.1.post6.dev5, 2.1.1.post6, 2.1.3.post1, 2.1.3.post2, 2.1.3.post3, 2.1.3.post4.dev2, 2.1.3.post4.dev5, 2.1.3.post4, 2.1.4.post1, 2.1.4.post2, 2.1.4.post3, 2.1.4.post4.dev2, 2.1.4.post4.dev7, 2.1.4.post4, 2.1.4.post5.dev4, 2.1.4.post5, 2.1.4.post6.dev1, 2.1.4.post6, 2.1.4.post7.dev1, 2.1.4.post7.dev2, 2.1.4.post7, 2.2.0.post1, 2.2.0.post2, 2.2.0.post3, 2.2.0.post4, 2.2.0.post5.dev3, 2.2.0.post5, 2.2.1.post1, 2.2.1.post2, 2.2.1.post3.dev2, 2.2.1.post3, 2.2.2.post1, 2.2.2.post2.dev2, 2.2.2.post2, 2.2.3.post1.dev918, 2.2.3.post1.dev919, 2.2.3.post1.dev924, 2.2.3.post1.dev929, 2.2.3.post1.dev942, 2.2.3.post1, 2.2.3.post2.dev1, 2.2.3.post2, 2.2.4.post1.dev1180, 2.2.4.post1.dev1190, 2.2.4.post1.dev1192, 2.2.4.post1, 2.2.4.post2.dev1, 2.2.4.post2, 2.2.4.post3.dev1, 2.2.4.post3, 2.2.4.post4.dev1, 2.2.4.post4.dev2, 2.2.4.post4, 2.2.5.post1.dev1264, 2.2.5.post1.dev1265, 2.2.5.post1, 2.2.5.post2, 2.2.5.post3.dev2, 2.3.0.dev20220323, 2.3.0.dev20220325, 2.3.0.dev20220326, 2.3.0.dev20220329, 2.3.0.dev20220330, 2.3.0.dev20220331, 2.3.0.dev20220401, 2.3.0.dev20220402, 2.3.0.dev20220405, 2.3.0.dev20220406, 2.3.0.dev20220407, 2.3.0.dev20220408, 2.3.0.dev20220409, 2.3.0.dev20220411, 2.3.0.dev20220412, 2.3.0.dev20220413, 2.3.0.dev20220414, 2.3.0.dev20220416, 2.3.0.dev20220419, 2.3.0.dev20220420, 2.3.0.dev20220421, 2.3.0.dev20220422, 2.3.0.post1.dev308, 2.3.0.post1.dev334, 2.3.0.post1.dev356)
ERROR: No matching distribution found for astronomer_certified[amazon,async,azure,cncf.kubernetes,elasticsearch,google,mysql,password,postgres,redis,slack,ssh,statsd,virtualenv]==2.3.0-1-*
WARNING: You are using pip version 21.2.4; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
The command '/bin/bash -o pipefail -e -u -x -c pip install "${AIRFLOW_MODULE}" celery flower         --constraint /tmp/build-time-pip-constraints.txt     && pip install "https://github.com/astronomer/astronomer-airflow-scripts/releases/download/v0.0.5/astronomer_airflow_scripts-0.0.5-py3-none-any.whl"     && pip install "astronomer-fab-security-manager==${ASTRONOMER_FAB_SECURITY_MANAGER_VERSION}"' returned a non-zero code: 1
dylanbstorey commented 2 years ago

Had a customer encounter this Monday as well.

https://github.com/astronomer/ap-airflow/blob/05568b66dee3c183441afc589174a848fcd8808e/2.2.5/bullseye/Dockerfile#L113. <- the version pinned here doesn't appear to be released yet per ticket 9180 on zendesk.

sfc-gh-cbaechtold commented 2 years ago

Is there any way to roll back via argument when running the docker build command? e.g. I have the previous image already built locally so can run no problem, but for lab participants, is there a way that they can run the docker build for the first time and rollback to the previously working version until the pinned version is released?

sfc-gh-cbaechtold commented 2 years ago

Current workaround is to pass VERSION=2.2.5-2 in as a build-arg, this works: DOCKER_BUILDKIT=0 docker build --build-arg PYTHON_MAJOR_MINOR_VERSION=3.8 --build-arg VERSION=2.2.5-2 -t ap-airflow:py38 https://github.com/astronomer/ap-airflow.git#master:2.2.5/bullseye

hwong557 commented 2 years ago

A similar problem exists in 2.3.3. The issue is line

https://github.com/astronomer/ap-airflow/blob/5a7a960da427700a87158e1247068bc58b5220e4/2.3.3/bullseye/Dockerfile#L113

asks Docker to install astronomer_certified at version 2.3.3-2-*, but no such version exists: (https://pip.astronomer.io/simple/astronomer-certified/). The solution above fixes it by passing --build-arg VERSION=2.3.3.* to docker build.