search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.01k stars 14.27k forks source link

Support `uv` in addition to `pip` when extending the Docker image #37785

Closed yehoshuadimarsky closed 8 months ago

yehoshuadimarsky commented 8 months ago

Description

I am trying to use the new phenomenal tool uv to add python packages on top of the base Airflow image. I kept running into the issue of uv not supporting non-virtual-envs, but this is now supported with version 0.1.12

Here is my sample Dockerfile. I cannot get it to work with either of these new flags: --python or --system.

FROM apache/airflow:slim-2.7.3-python3.10

ENV PIP_DISABLE_PIP_VERSION_CHECK=1
ENV PIP_ROOT_USER_ACTION="ignore"

USER root

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
        build-essential libmariadb-dev \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

USER airflow

RUN pip install "uv==0.1.12"

COPY requirements.txt /
COPY constraints.txt /

# with --python
RUN uv pip install --no-cache -r /requirements.txt -c /constraints.txt --python $(which python)

# with --system
# RUN uv pip install --no-cache -r /requirements.txt -c /constraints.txt --system

In both cases, I get this error:

=> ERROR [6/6] RUN uv pip install --no-cache -r /requirements.txt -c /constraints.txt --python $(which python  18.5s
------                                                                                                                
 > [6/6] RUN uv pip install --no-cache -r /requirements.txt -c /constraints.txt --python $(which python):             
4.061 Resolved 239 packages in 3.48s                                                                                  
16.86 Downloaded 238 packages in 12.79s                                                                               
18.07 error: failed to remove file `/usr/local/lib/python3.10/site-packages/_distutils_hack/__init__.py`
18.07   Caused by: Permission denied (os error 13)
------
Dockerfile.airflow2:22
--------------------
  20 |     COPY constraints.txt /
  21 |     
  22 | >>> RUN uv pip install --no-cache -r /requirements.txt -c /constraints.txt --python $(which python)
  23 |     
  24 |     # RUN uv pip install --no-cache -r /requirements.txt -c /constraints.txt --system
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c uv pip install --no-cache -r /requirements.txt -c /constraints.txt --python $(which python)" did not complete successfully: exit code: 2

I thought, ok, let me run it as USER root, but then I get an error about installing pip packages as root, from here: https://airflow.apache.org/docs/docker-stack/build.html#adding-new-pypi-packages-individually

Any suggestions on using uv with the Airflow Docker image?

Thanks

Use case/motivation

No response

Related issues

I see that Airflow has already adopted uv internally for its CI here and see (https://github.com/apache/airflow/issues?q=uv), so I think this should be doable for the external Docker image as well

Are you willing to submit a PR?

Code of Conduct

potiuk commented 8 months ago

Yes. We just switched to uv for CI images: https://github.com/apache/airflow/pull/37692 and indeed uv is phenomenal. It saves as 50% - 60% of image building time in some cases. Yes. I absolutely plan to have uv support and yes, I was waiting for it to support non-virtualenv settings, so that might be the right time to attempt it :)

potiuk commented 8 months ago

BTW. It's not going to be very easy (and might need extra feature in uv because we are using --user installation in PROD image to isolate system from user dependencies. i will open an issue for UV about it

potiuk commented 8 months ago

Opened an issue for that https://github.com/astral-sh/uv/issues/2077

potiuk commented 8 months ago

Found a way around it. For Airflow 2.9, uv should be fully supported - https://github.com/apache/airflow/pull/37796 regardless if uv flag will be supported or not by uv. @yehoshuadimarsky -> can you please test if that solves your problem?

The way how to test it:

export AIRFLOW_VERSION=2.8.2
export DOCKER_BUILDKIT=1

docker build . \
    --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --tag "my-tag:0.0.1"

This should be built by pip - but when you get the image, you can extend it - latest uv version should be already in and should work without the need of adding any flags.

You can also make another test:

export AIRFLOW_VERSION=2.8.2
export DOCKER_BUILDKIT=1

docker build . \
    --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --build-arg AIRFLOW_USE_UV="true" \
    --tag "my-tag:0.0.1"

Should use uv to build the docker image. That one (according to my tests) should be 50%-60% faster to build from scratch comparing to the pip one. We do not want to enable it by default, but it should be fully usable and working.

Can you please test and give feedback @yehoshuadimarsky ?

yehoshuadimarsky commented 8 months ago

to be honest i'm not fully comfortable building the docker image using the source "code" Dockerfile in your PR, it's a bit long and overwhelming (>1500 line file). I'm in no rush, more of an enthusiast wanting to try out cool new tools, this is not a hard blocker for me now. I'm ok waiting until you finalize the workflow and have a finished image built.

potiuk commented 8 months ago

to be honest i'm not fully comfortable building the docker image using the source "code" Dockerfile in your PR, it's a bit long and overwhelming (>1500 line file). I'm in no rush, more of an enthusiast wanting to try out cool new tools, this is not a hard blocker for me now. I'm ok waiting until you finalize the workflow and have a finished image built.

To be honest, there are good instructoins to run the build. It's exactly one Dockerfile to copy and single instruction to build it :) - and without people like you testihg it before I can only have hopes that it will solve your problem, but never a certainty. testing and trying before merge is the best way to see. If your tests confirm that it works, we can merge it with confidence, if it does not, then I have a chance to fix it before, it's of course up to you, but if it does not fit your needs after we release it in 2.9 - you have now opportunity to prevent it. You can make use of it or not, but then it it does not, then you might have only self-complaint that you have not tested it.

But of course - it's up to you to help to test it or not.

potiuk commented 8 months ago

(and if it does not work we will find out about month from now - when 2.9 will be released BTW).

yehoshuadimarsky commented 8 months ago

Just tried building the image on Windows following these steps:

  1. Cloned this repo
  2. Checkout the PR branch https://github.com/apache/airflow/pull/37796
  3. Ran in Powershell: 
    
$env:AIRFLOW_VERSION="2.8.2"
$env:DOCKER_BUILDKIT=1

docker build . --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" --tag "my-tag:0.0.1"

4. But I got this error

<details>

[+] Building 10.2s (44/73) docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 70.84kB 0.0s => resolve image config for docker.io/docker/dockerfile:1.4 0.9s => docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.8s => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB 0.4s => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB 0.0s => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B 0.0s => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB 0.0s => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 0.3s => [internal] load .dockerignore 0.0s => => transferring context: 3.21kB 0.0s => [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.5s => [internal] preparing inline document 0.1s => [internal] settings cache mount permissions 0.2s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.3s => [internal] load build context 0.3s => => transferring context: 70.84kB 0.0s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.3s => [airflow-build-image 1/16] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 6.7s => => resolve docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 0.1s => => sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 1.86kB / 1.86kB 0.0s => => sha256:3e9292a5e3bc66896a7c1dd89d59626123fbc68f9c8d45db96eacd53b0ad4580 1.37kB / 1.37kB 0.0s => => sha256:809c394c4cffdc8d84af5418b1ecf9c3195b6b9b9b3412690951f30677727a56 6.97kB / 6.97kB 0.0s => => sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56 29.12MB / 29.12MB 1.5s => => sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d 3.51MB / 3.51MB 0.4s => => sha256:07b545b886b2fba30f518b093792e37c6cd9cf02468897770202912012325b53 13.75MB / 13.75MB 1.1s => => sha256:f86bc27bff6164ce6b0a86a633bcd9640ff7467bc1f5e4eb824411ac270ed5ca 243B / 243B 0.5s => => sha256:1a56bca2cd81c555af1702d6725f31abff239026723801d1f5adf4aad7c5e14c 3.13MB / 3.13MB 0.8s => => extracting sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56 2.8s => => extracting sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d 0.3s => => extracting sha256:07b545b886b2fba30f518b093792e37c6cd9cf02468897770202912012325b53 1.1s => => extracting sha256:f86bc27bff6164ce6b0a86a633bcd9640ff7467bc1f5e4eb824411ac270ed5ca 0.0s => => extracting sha256:1a56bca2cd81c555af1702d6725f31abff239026723801d1f5adf4aad7c5e14c 0.5s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.0s => [scripts 1/17] COPY <<EOF /install_os_dependencies.sh 0.3s => [scripts 2/17] COPY <<EOF /install_mysql.sh 0.0s => [scripts 3/17] COPY <<EOF /install_mssql.sh 0.0s => [scripts 4/17] COPY <<EOF /install_postgres.sh 0.0s => [scripts 5/17] COPY <<EOF /install_packaging_tools.sh 0.0s => [scripts 6/17] COPY <<EOF /install_airflow_dependencies_from_branch_tip.sh 0.0s => [scripts 7/17] COPY <<EOF /common.sh 0.0s => [scripts 8/17] COPY <<EOF /pip 0.0s => [scripts 9/17] COPY <<EOF /install_from_docker_context_files.sh 0.0s => [scripts 10/17] COPY <<EOF /get_package_specs.py 0.0s => [scripts 11/17] COPY <<EOF /install_airflow.sh 0.0s => [scripts 12/17] COPY <<EOF /install_additional_dependencies.sh 0.0s => [scripts 13/17] COPY <<EOF /create_prod_venv.sh 0.0s => [scripts 14/17] COPY <<EOF /create_prod_venv.sh 0.0s => [scripts 15/17] COPY <<EOF /entrypoint_prod.sh 0.1s => [scripts 16/17] COPY <<EOF /clean-logs.sh 0.0s => [scripts 17/17] COPY <<EOF /airflow-scheduler-autorestart.sh 0.0s => [airflow-build-image 2/16] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.1s => ERROR [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 0.7s => ERROR [main 3/19] RUN bash /scripts/docker/install_os_dependencies.sh runtime 0.7s

[airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev: : invalid option name/install_os_dependencies.sh: line 2: set: pipefail

[main 3/19] RUN bash /scripts/docker/install_os_dependencies.sh runtime: : invalid option name/install_os_dependencies.sh: line 2: set: pipefail

Dockerfile:1338

1336 | 1337 | COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 1338 | >>> RUN bash /scripts/docker/install_os_dependencies.sh dev 1339 | 1340 | ARG INSTALL_MYSQL_CLIENT="true"

ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c bash /scripts/docker/install_os_dependencies.sh dev" did not complete successfully: exit code: 2

View build details: docker-desktop://dashboard/build/default/default/dpgt50krcuoyudg0qo6mv8mte


</details>

Is this because I'm on Windows not Unix?

**Edit:** So tried using GitHub Codespaces, got this:

root@53324cd94090:/opt/airflow# which docker /usr/bin/docker root@53324cd94090:/opt/airflow# export AIRFLOW_VERSION=2.8.2 export DOCKER_BUILDKIT=1 root@53324cd94090:/opt/airflow# docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \ --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \ --tag "my-tag:0.0.1" ERROR: BuildKit is enabled but the buildx component is missing or broken. Install the buildx component to build images with BuildKit: https://docs.docker.com/go/buildx/ root@53324cd94090:/opt/airflow#

yehoshuadimarsky commented 8 months ago

Edit #3: Followed instructions here https://docs.docker.com/engine/install/debian/ to setup Docker with BuildKit on GH Codespaces, now I can build the image. It ran for a while, but then failed with this output:

``` root@53324cd94090:/opt/airflow# docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \ --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \ --tag "my-tag:0.0.1" [+] Building 268.2s (61/75) docker:default => [internal] load .dockerignore 0.3s => => transferring context: 3.08kB 0.0s => [internal] load build definition from Dockerfile 0.3s => => transferring dockerfile: 69.12kB 0.0s => resolve image config for docker.io/docker/dockerfile:1.4 0.5s => [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s => docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.7s => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.1s => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB 0.0s => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B 0.0s => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB 0.0s => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB 0.2s => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 0.2s => [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.6s => [auth] library/python:pull token for registry-1.docker.io 0.0s => [internal] preparing inline document 0.7s => [internal] preparing inline document 1.7s => [internal] preparing inline document 1.3s => [internal] preparing inline document 1.9s => [internal] preparing inline document 1.5s => [internal] preparing inline document 1.9s => [internal] preparing inline document 1.6s => [internal] preparing inline document 1.9s => [internal] settings cache mount permissions 2.4s => [internal] preparing inline document 2.0s => [internal] preparing inline document 2.0s => [internal] preparing inline document 2.1s => [internal] preparing inline document 2.2s => [internal] load build context 1.0s => => transferring context: 69.12kB 0.0s => [internal] preparing inline document 2.2s => [main 1/19] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 3.2s => => resolve docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 1.2s => => sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 1.86kB / 1.86kB 0.0s => => sha256:3e9292a5e3bc66896a7c1dd89d59626123fbc68f9c8d45db96eacd53b0ad4580 1.37kB / 1.37kB 0.0s => => sha256:809c394c4cffdc8d84af5418b1ecf9c3195b6b9b9b3412690951f30677727a56 6.97kB / 6.97kB 0.0s => [internal] preparing inline document 2.3s => [internal] preparing inline document 2.3s => [internal] preparing inline document 0.2s => [scripts 1/17] COPY < [scripts 2/17] COPY < [scripts 3/17] COPY < [scripts 4/17] COPY < [scripts 5/17] COPY < [scripts 6/17] COPY < [scripts 7/17] COPY < [scripts 8/17] COPY < [scripts 9/17] COPY < [scripts 10/17] COPY < [scripts 11/17] COPY < [scripts 12/17] COPY < [scripts 13/17] COPY < [scripts 14/17] COPY < [scripts 15/17] COPY < [scripts 16/17] COPY < [scripts 17/17] COPY < [main 2/19] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.3s => [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 170.5s => [main 3/19] RUN bash /scripts/docker/install_os_dependencies.sh runtime 67.6s => [main 4/19] COPY --from=scripts common.sh /scripts/docker/ 0.4s => [main 5/19] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.5s => [main 6/19] RUN bash /scripts/docker/install_mysql.sh prod && bash /scripts/docker/install_mssql.sh prod && bash /scripts/docker/install_postgres.sh prod && adduser --gecos "First Last,RoomNumber,WorkPho 33.0s => [airflow-build-image 4/16] COPY --from=scripts common.sh /scripts/docker/ 0.3s => [airflow-build-image 5/16] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.3s => [airflow-build-image 6/16] RUN bash /scripts/docker/install_mysql.sh dev && bash /scripts/docker/install_mssql.sh dev && bash /scripts/docker/install_postgres.sh dev 25.8s => [airflow-build-image 7/16] RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password --quiet "airflow" --uid "50000" --gid "0" --home "/home/airflow" && mkdir -p /opt/airflow && c 1.2s => [airflow-build-image 8/16] COPY --chown=50000:0 Dockerfile /docker-context-files 0.3s => [airflow-build-image 9/16] RUN if [[ -f /docker-context-files/pip.conf ]]; then mkdir -p /home/airflow/.config/pip; cp /docker-context-files/pip.conf "/home/airflow/.config/pip/pip.conf"; fi; 0.5s => [airflow-build-image 10/16] COPY --from=scripts common.sh install_packaging_tools.sh install_airflow_dependencies_from_branch_tip.sh create_prod_venv.sh /scripts/docker/ 0.5s => [airflow-build-image 11/16] RUN bash /scripts/docker/install_packaging_tools.sh; bash /scripts/docker/create_prod_venv.sh; if [[ false == "true" && false == "false" && false == "false" ]]; the 14.0s => [airflow-build-image 12/16] COPY --chown=airflow:0 Dockerfile /Dockerfile 0.3s => [airflow-build-image 13/16] WORKDIR /opt/airflow 0.3s => [airflow-build-image 14/16] COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh install_additional_dependencies.sh create_prod_venv.sh get_package_specs.py /scripts/docker/ 0.4s => ERROR [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-amd64-9,target=/tmp/.cache/pip,uid=50000 if [[ false == "true" ]]; then bash /scripts/docker/install_from_docker_c 43.6s ------ > [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-amd64-9,target=/tmp/.cache/pip,uid=50000 if [[ false == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find "/home/airflow/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find "/home/airflow/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find "/home/airflow/.local" -executable ! -type l -print0 | xargs --null chmod g+x; find "/home/airflow/.local" ! -type l -print0 | xargs --null chmod g+rw: 0.722 0.722 Using 'uv' to install Airflow 0.722 0.736 % Total % Received % Xferd Average Speed Time Time Time Current 0.737 Dload Upload Total Spent Left Speed 100 18207 100 18207 0 0 143k 0 --:--:-- --:--:-- --:--:-- 144k 0.864 PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/mssql-tools/bin 0.864 Using 'uv' to install Airflow 0.866 uv on path: /home/airflow/.local/bin/uv 0.872 Using uv: uv 0.1.13 0.873 + uv pip install 'apache-airflow[aiobotocore,amazon,async,celery,cncf-kubernetes,common-io,docker,elasticsearch,ftp,google,google-auth,graphviz,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,uv,virtualenv]==2.8.2' --constraint /home/airflow/constraints.txt 0.873 0.873 Installing all packages with constraints and upgrade if needed 0.873 7.668 Resolved 350 packages in 6.66s 29.86 Downloaded 350 packages in 22.17s 32.07 Installed 350 packages in 2.18s 32.07 + adal==1.2.7 32.07 + adlfs==2024.2.0 32.07 + aiobotocore==2.11.2 32.07 + aiofiles==23.2.1 32.07 + aiohttp==3.9.3 32.07 + aioitertools==0.11.0 32.07 + aiosignal==1.3.1 32.07 + alembic==1.13.1 32.07 + amqp==5.2.0 32.07 + anyio==4.3.0 32.07 + apache-airflow==2.8.2 32.07 + apache-airflow-providers-amazon==8.18.0 32.07 + apache-airflow-providers-celery==3.6.0 32.07 + apache-airflow-providers-cncf-kubernetes==8.0.0 32.07 + apache-airflow-providers-common-io==1.3.0 32.07 + apache-airflow-providers-common-sql==1.11.0 32.07 + apache-airflow-providers-docker==3.9.1 32.07 + apache-airflow-providers-elasticsearch==5.3.3 32.07 + apache-airflow-providers-ftp==3.7.0 32.07 + apache-airflow-providers-google==10.15.0 32.07 + apache-airflow-providers-grpc==3.4.1 32.07 + apache-airflow-providers-hashicorp==3.6.3 32.07 + apache-airflow-providers-http==4.9.1 32.07 + apache-airflow-providers-imap==3.5.0 32.07 + apache-airflow-providers-microsoft-azure==9.0.0 32.07 + apache-airflow-providers-mysql==5.5.3 32.07 + apache-airflow-providers-odbc==4.4.1 32.07 + apache-airflow-providers-openlineage==1.5.0 32.07 + apache-airflow-providers-postgres==5.10.1 32.07 + apache-airflow-providers-redis==3.6.0 32.07 + apache-airflow-providers-sendgrid==3.4.0 32.07 + apache-airflow-providers-sftp==4.9.0 32.07 + apache-airflow-providers-slack==8.6.1 32.07 + apache-airflow-providers-snowflake==5.3.1 32.07 + apache-airflow-providers-sqlite==3.7.1 32.08 + apache-airflow-providers-ssh==3.10.1 32.08 + apispec==6.4.0 32.08 + argcomplete==3.2.2 32.08 + asgiref==3.7.2 32.08 + asn1crypto==1.5.1 32.08 + async-timeout==4.0.3 32.08 + asyncssh==2.14.2 32.08 + attrs==23.2.0 32.08 + authlib==1.3.0 32.08 + azure-batch==14.1.0 32.08 + azure-common==1.1.28 32.08 + azure-core==1.30.0 32.08 + azure-cosmos==4.5.1 32.08 + azure-datalake-store==0.0.53 32.08 + azure-identity==1.15.0 32.08 + azure-keyvault-secrets==4.8.0 32.08 + azure-kusto-data==4.3.1 32.08 + azure-mgmt-containerinstance==10.1.0 32.08 + azure-mgmt-containerregistry==10.3.0 32.08 + azure-mgmt-core==1.4.0 32.08 + azure-mgmt-cosmosdb==9.4.0 32.08 + azure-mgmt-datafactory==5.0.0 32.08 + azure-mgmt-datalake-nspkg==3.0.1 32.08 + azure-mgmt-datalake-store==0.5.0 32.08 + azure-mgmt-nspkg==3.0.2 32.08 + azure-mgmt-resource==23.0.1 32.08 + azure-mgmt-storage==21.1.0 32.08 + azure-nspkg==3.0.2 32.08 + azure-servicebus==7.11.4 32.08 + azure-storage-blob==12.19.0 32.08 + azure-storage-file-datalake==12.14.0 32.08 + azure-storage-file-share==12.15.0 32.08 + azure-synapse-artifacts==0.18.0 32.08 + azure-synapse-spark==0.7.0 32.08 + babel==2.14.0 32.08 + backoff==2.2.1 32.08 + backports-zoneinfo==0.2.1 32.08 + bcrypt==4.1.2 32.08 + beautifulsoup4==4.12.3 32.08 + billiard==4.2.0 32.08 + blinker==1.7.0 32.08 + boto3==1.33.13 32.08 + botocore==1.33.13 32.08 + cachelib==0.9.0 32.08 + cachetools==5.3.2 32.08 + cattrs==23.2.3 32.08 + celery==5.3.6 32.08 + certifi==2024.2.2 32.08 + cffi==1.16.0 32.08 + chardet==5.2.0 32.08 + charset-normalizer==3.3.2 32.08 + click==8.1.7 32.08 + click-didyoumean==0.3.0 32.08 + click-plugins==1.1.1 32.08 + click-repl==0.3.0 32.08 + clickclick==20.10.2 32.08 + colorama==0.4.6 32.08 + colorlog==4.8.0 32.08 + configupdater==3.2 32.08 + connexion==2.14.2 32.08 + cron-descriptor==1.4.3 32.08 + croniter==2.0.1 32.08 + cryptography==41.0.7 32.08 + db-dtypes==1.2.0 32.08 + decorator==5.1.1 32.08 + deprecated==1.2.14 32.08 + dill==0.3.1.1 32.08 + distlib==0.3.8 32.08 + dnspython==2.6.1 32.08 + docker==7.0.0 32.08 + docutils==0.20.1 32.08 + elastic-transport==8.12.0 32.08 + elasticsearch==8.12.1 32.08 + email-validator==1.3.1 32.08 + eventlet==0.35.2 32.08 + exceptiongroup==1.2.0 32.08 + filelock==3.13.1 32.08 + flask==2.2.5 32.08 + flask-appbuilder==4.3.11 32.08 + flask-babel==2.0.0 32.08 + flask-caching==2.1.0 32.08 + flask-jwt-extended==4.6.0 32.08 + flask-limiter==3.5.1 32.08 + flask-login==0.6.3 32.08 + flask-session==0.5.0 32.08 + flask-sqlalchemy==2.5.1 32.08 + flask-wtf==1.2.1 32.08 + flower==2.0.1 32.08 + frozenlist==1.4.1 32.08 + fsspec==2024.2.0 32.08 + gcloud-aio-auth==4.2.3 32.08 + gcloud-aio-bigquery==7.1.0 32.08 + gcloud-aio-storage==9.2.0 32.08 + gcsfs==2024.2.0 32.08 + gevent==24.2.1 32.08 + google-ads==23.1.0 32.08 + google-analytics-admin==0.22.6 32.08 + google-api-core==2.17.1 32.08 + google-api-python-client==2.119.0 32.08 + google-auth==2.28.1 32.08 + google-auth-httplib2==0.2.0 32.08 + google-auth-oauthlib==1.2.0 32.08 + google-cloud-aiplatform==1.42.1 32.08 + google-cloud-appengine-logging==1.4.2 32.08 + google-cloud-audit-log==0.2.5 32.08 + google-cloud-automl==2.13.2 32.08 + google-cloud-batch==0.17.12 32.08 + google-cloud-bigquery==3.17.2 32.08 + google-cloud-bigquery-datatransfer==3.15.0 32.08 + google-cloud-bigquery-storage==2.24.0 32.08 + google-cloud-bigtable==2.23.0 32.08 + google-cloud-build==3.23.2 32.08 + google-cloud-compute==1.17.0 32.08 + google-cloud-container==2.41.0 32.08 + google-cloud-core==2.4.1 32.08 + google-cloud-datacatalog==3.18.2 32.08 + google-cloud-dataflow-client==0.8.9 32.08 + google-cloud-dataform==0.5.8 32.08 + google-cloud-dataplex==1.12.2 32.08 + google-cloud-dataproc==5.9.2 32.08 + google-cloud-dataproc-metastore==1.15.2 32.08 + google-cloud-dlp==3.15.2 32.08 + google-cloud-kms==2.21.2 32.08 + google-cloud-language==2.13.2 32.08 + google-cloud-logging==3.9.0 32.08 + google-cloud-memcache==1.9.2 32.08 + google-cloud-monitoring==2.19.2 32.08 + google-cloud-orchestration-airflow==1.12.0 32.08 + google-cloud-os-login==2.14.2 32.08 + google-cloud-pubsub==2.19.6 32.08 + google-cloud-redis==2.15.2 32.08 + google-cloud-resource-manager==1.12.2 32.08 + google-cloud-run==0.10.4 32.08 + google-cloud-secret-manager==2.18.2 32.08 + google-cloud-spanner==3.42.0 32.08 + google-cloud-speech==2.25.0 32.08 + google-cloud-storage==2.14.0 32.08 + google-cloud-storage-transfer==1.11.2 32.08 + google-cloud-tasks==2.16.2 32.08 + google-cloud-texttospeech==2.16.2 32.08 + google-cloud-translate==3.15.2 32.08 + google-cloud-videointelligence==2.13.2 32.08 + google-cloud-vision==3.7.1 32.08 + google-cloud-workflows==1.14.2 32.08 + google-crc32c==1.5.0 32.08 + google-re2==1.1 32.08 + google-resumable-media==2.7.0 32.09 + googleapis-common-protos==1.62.0 32.09 + graphviz==0.20.1 32.09 + greenlet==3.0.3 32.09 + grpc-google-iam-v1==0.13.0 32.09 + grpc-interceptor==0.15.4 32.09 + grpcio==1.62.0 32.09 + grpcio-gcp==0.2.2 32.09 + grpcio-status==1.62.0 32.09 + gunicorn==21.2.0 32.09 + h11==0.14.0 32.09 + httpcore==0.16.3 32.09 + httplib2==0.22.0 32.09 + httpx==0.23.3 32.09 + humanize==4.9.0 32.09 + hvac==2.1.0 32.09 + idna==3.6 32.09 + ijson==3.2.3 32.09 + importlib-metadata==6.11.0 32.09 + importlib-resources==5.13.0 32.09 + inflection==0.5.1 32.09 + isodate==0.6.1 32.09 + itsdangerous==2.1.2 32.09 + jinja2==3.1.3 32.09 + jmespath==0.10.0 32.09 + json-merge-patch==0.2 32.09 + jsonpath-ng==1.6.1 32.09 + jsonschema==4.21.1 32.09 + jsonschema-specifications==2023.12.1 32.09 + kombu==5.3.5 32.09 + kubernetes==29.0.0 32.09 + kubernetes-asyncio==29.0.0 32.09 + lazy-object-proxy==1.10.0 32.09 + ldap3==2.9.1 32.09 + limits==3.9.0 32.09 + linkify-it-py==2.0.3 32.09 + lockfile==0.12.2 32.09 + looker-sdk==24.2.0 32.09 + lxml==5.1.0 32.09 + mako==1.3.2 32.09 + markdown-it-py==3.0.0 32.09 + markupsafe==2.1.5 32.09 + marshmallow==3.20.2 32.09 + marshmallow-oneofschema==3.1.1 32.09 + marshmallow-sqlalchemy==0.26.1 32.09 + mdit-py-plugins==0.4.0 32.09 + mdurl==0.1.2 32.09 + more-itertools==10.2.0 32.09 + msal==1.27.0 32.09 + msal-extensions==1.1.0 32.09 + msrest==0.7.1 32.09 + msrestazure==0.6.4 32.09 + multidict==6.0.5 32.09 + mysql-connector-python==8.3.0 32.09 + mysqlclient==2.2.4 32.09 + numpy==1.24.4 32.09 + oauthlib==3.2.2 32.09 + openlineage-integration-common==1.9.0 32.09 + openlineage-python==1.9.0 32.09 + openlineage-sql==1.9.0 32.09 + opentelemetry-api==1.23.0 32.09 + opentelemetry-exporter-otlp==1.23.0 32.09 + opentelemetry-exporter-otlp-proto-common==1.23.0 32.09 + opentelemetry-exporter-otlp-proto-grpc==1.23.0 32.09 + opentelemetry-exporter-otlp-proto-http==1.23.0 32.09 + opentelemetry-proto==1.23.0 32.09 + opentelemetry-sdk==1.23.0 32.09 + opentelemetry-semantic-conventions==0.44b0 32.09 + ordered-set==4.1.0 32.09 + packaging==23.2 32.09 + pandas==2.0.3 32.09 + pandas-gbq==0.21.0 32.09 + paramiko==3.4.0 32.09 + pathspec==0.12.1 32.09 + pendulum==3.0.0 32.09 + pkgutil-resolve-name==1.3.10 32.09 + platformdirs==3.11.0 32.09 + pluggy==1.4.0 32.09 + ply==3.11 32.09 + portalocker==2.8.2 32.09 + prison==0.2.1 32.09 + prometheus-client==0.20.0 32.09 + prompt-toolkit==3.0.43 32.09 + proto-plus==1.23.0 32.09 + protobuf==4.25.3 32.09 + psutil==5.9.8 32.09 + psycopg2-binary==2.9.9 32.09 + pyarrow==14.0.2 32.09 + pyasn1==0.5.1 32.09 + pyasn1-modules==0.3.0 32.09 + pyathena==3.3.0 32.09 + pycparser==2.21 32.09 + pydata-google-auth==1.8.2 32.09 + pygments==2.17.2 32.09 + pyjwt==2.8.0 32.09 + pynacl==1.5.0 32.09 + pyodbc==5.1.0 32.09 + pyopenssl==24.0.0 32.09 + pyparsing==3.1.1 32.09 + python-daemon==3.0.1 32.09 + python-dateutil==2.8.2 32.09 + python-dotenv==1.0.1 32.09 + python-http-client==3.3.7 32.09 + python-ldap==3.4.4 32.09 + python-nvd3==0.15.0 32.09 + python-slugify==8.0.4 32.09 + pytz==2024.1 32.09 + pyyaml==6.0.1 32.09 + redis==4.6.0 32.09 + redshift-connector==2.1.0 32.09 + referencing==0.33.0 32.09 + requests==2.31.0 32.09 + requests-oauthlib==1.3.1 32.09 + requests-toolbelt==1.0.0 32.09 + rfc3339-validator==0.1.4 32.09 + rfc3986==1.5.0 32.09 + rich==13.7.0 32.09 + rich-argparse==1.4.0 32.09 + rpds-py==0.18.0 32.09 + rsa==4.9 32.09 + s3transfer==0.8.2 32.09 + scramp==1.4.4 32.09 + sendgrid==6.11.0 32.09 + setproctitle==1.3.3 32.09 - setuptools==56.0.0 32.09 + setuptools==66.1.1 32.09 + shapely==2.0.3 32.09 + six==1.16.0 32.09 + slack-sdk==3.27.0 32.09 + sniffio==1.3.0 32.09 + snowflake-connector-python==3.7.1 32.09 + snowflake-sqlalchemy==1.5.1 32.09 + sortedcontainers==2.4.0 32.09 + soupsieve==2.5 32.09 + sqlalchemy==1.4.51 32.09 + sqlalchemy-bigquery==1.9.0 32.09 + sqlalchemy-jsonfield==1.0.2 32.09 + sqlalchemy-redshift==0.8.14 32.09 + sqlalchemy-spanner==1.6.2 32.09 + sqlalchemy-utils==0.41.1 32.09 + sqlparse==0.4.4 32.09 + sshtunnel==0.4.0 32.09 + starkbank-ecdsa==2.2.0 32.09 + statsd==4.0.1 32.09 + tabulate==0.9.0 32.09 + tenacity==8.2.3 32.09 + termcolor==2.4.0 32.09 + text-unidecode==1.3 32.09 + time-machine==2.13.0 32.09 + tomlkit==0.12.3 32.09 + tornado==6.4 32.09 + typing-extensions==4.9.0 32.09 + tzdata==2024.1 32.09 + uc-micro-py==1.0.3 32.09 + unicodecsv==0.14.1 32.09 + universal-pathlib==0.1.4 32.09 + uritemplate==4.1.1 32.09 + urllib3==1.26.18 32.09 + vine==5.1.0 32.09 + virtualenv==20.25.1 32.09 + watchtower==3.0.1 32.09 + wcwidth==0.2.13 32.09 + websocket-client==1.7.0 32.09 + werkzeug==2.2.3 32.09 + wrapt==1.16.0 32.09 + wtforms==3.1.2 32.09 + yarl==1.9.4 32.09 + zipp==3.17.0 32.09 + zope-event==5.0 32.09 + zope-interface==6.2 32.11 + common::install_packaging_tools 32.11 32.11 Installing pip version 24.0 32.11 32.11 + echo 32.11 + echo 'Installing pip version 24.0' 32.11 + echo 32.11 + [[ 24.0 =~ .*https.* ]] 32.12 + pip install --root-user-action ignore --disable-pip-version-check pip==24.0 33.10 Requirement already satisfied: pip==24.0 in /home/airflow/.local/lib/python3.8/site-packages (24.0) 37.14 + echo 37.14 37.14 + echo 'Installing uv version 0.1.13' 37.14 Installing uv version 0.1.13 37.14 + echo 37.14 37.14 + [[ 0.1.13 =~ .*https.* ]] 37.14 + pip install --root-user-action ignore --disable-pip-version-check uv==0.1.13 38.11 Requirement already satisfied: uv==0.1.13 in /home/airflow/.local/lib/python3.8/site-packages (0.1.13) 41.80 + mkdir -p /home/airflow/.local/bin 41.81 + uv pip install --upgrade --resolution lowest-direct 'apache-airflow[aiobotocore,amazon,async,celery,cncf-kubernetes,common-io,docker,elasticsearch,ftp,google,google-auth,graphviz,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,uv,virtualenv]==2.8.2' 43.06 error: Failed to download and build: apache-airflow==1.8.1 43.06 Caused by: Failed to build: apache-airflow==1.8.1 43.07 Caused by: Build backend failed to determine extra requires with `build_wheel()`: 43.07 --- stdout: 43.07 43.07 --- stderr: 43.07 Traceback (most recent call last): 43.07 File "", line 14, in 43.07 File "/home/airflow/.cache/uv/.tmplTHsNi/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel 43.07 return self._get_build_requires(config_settings, requirements=['wheel']) 43.07 File "/home/airflow/.cache/uv/.tmplTHsNi/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires 43.07 self.run_setup() 43.07 File "/home/airflow/.cache/uv/.tmplTHsNi/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup 43.07 super().run_setup(setup_script=setup_script) 43.07 File "/home/airflow/.cache/uv/.tmplTHsNi/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup 43.07 exec(code, locals()) 43.07 File "", line 112 43.07 async = [ 43.07 ^ 43.07 SyntaxError: invalid syntax 43.07 --- ------ Dockerfile:1520 -------------------- 1519 | # hadolint ignore=SC2086, SC2010, DL3042 1520 | >>> RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \ 1521 | >>> if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \ 1522 | >>> bash /scripts/docker/install_from_docker_context_files.sh; \ 1523 | >>> fi; \ 1524 | >>> if ! airflow version 2>/dev/null >/dev/null; then \ 1525 | >>> bash /scripts/docker/install_airflow.sh; \ 1526 | >>> fi; \ 1527 | >>> if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \ 1528 | >>> bash /scripts/docker/install_additional_dependencies.sh; \ 1529 | >>> fi; \ 1530 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; \ 1531 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; \ 1532 | >>> # make sure that all directories and files in .local are also group accessible 1533 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" -executable ! -type l -print0 | xargs --null chmod g+x; \ 1534 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" ! -type l -print0 | xargs --null chmod g+rw 1535 | -------------------- ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == \"true\" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n \"${ADDITIONAL_PYTHON_DEPS}\" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local\" -executable ! -type l -print0 | xargs --null chmod g+x; find \"${AIRFLOW_USER_HOME_DIR}/.local\" ! -type l -print0 | xargs --null chmod g+rw" did not complete successfully: exit code: 2 ```
potiuk commented 8 months ago

1.git config --global core.autocrlf should fix windows 2.3 -> updated default. in the new version now.. Seems uv was set as default to build image but still has some teething problems. The update show switch image building to pip but uv should work now.

potiuk commented 8 months ago

All right - I did more testing and I think It should work nicer now :)

potiuk commented 8 months ago

https://github.com/apache/airflow/pull/37796

yehoshuadimarsky commented 8 months ago

Just tried again, this time on my M1 Macbook (2020 Pro), doesn't work:

 ~/Documents/GitHub/ git clone git@github.com:apache/airflow.git airflow-upstream
Cloning into 'airflow-upstream'...
remote: Enumerating objects: 400720, done.
remote: Counting objects: 100% (662/662), done.
remote: Compressing objects: 100% (410/410), done.
remote: Total 400720 (delta 282), reused 573 (delta 244), pack-reused 400058
Receiving objects: 100% (400720/400720), 258.46 MiB | 10.83 MiB/s, done.
Resolving deltas: 100% (311879/311879), done.
 ~/Documents/GitHub/ cd airflow-upstream
 ~/Documents/GitHub/airflow-upstream/ [main] git checkout use-uv-for-prod-image
branch 'use-uv-for-prod-image' set up to track 'origin/use-uv-for-prod-image'.
Switched to a new branch 'use-uv-for-prod-image'

Here's the command and the error output:

```zsh  ~/Documents/GitHub/airflow-upstream/ [use-uv-for-prod-image] export AIRFLOW_VERSION=2.8.2 export DOCKER_BUILDKIT=1 docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \ --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \ --tag "my-tag:0.0.1" [+] Building 75.3s (59/72) docker:desktop-linux => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 72.08kB 0.0s => resolve image config for docker.io/docker/dockerfile:1.4 0.5s => docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s => => sha256:8d5998862fafcacec68298ab34c9c28973251a5082836f31de0188cd914fae33 2.36kB / 2.36kB 0.0s => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB 0.0s => => sha256:f259fd86ee23d7b4a2944432e3a09f45fa288bb84bfbd222b19b2abc7ddffbb9 528B / 528B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 3.12kB 0.0s => [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.5s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.0s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [main 1/18] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 4.1s => => resolve docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 0.1s => => sha256:515463c9b88285509388c9d127a8a56cf9602cd56ce59356c283b978897d7104 1.37kB / 1.37kB 0.0s => => sha256:054df0836f5230590646d8b7c62c54009a80c5c95c0c79e5430c3ef0f9c870a7 6.98kB / 6.98kB 0.0s => => sha256:f546e941f15b76df3d982d56985432b05bc065e3923fb35be25a4d33d5c0f911 29.16MB / 29.16MB 1.7s => => sha256:24935aba99a712c5a6efc86118762b0f914e1577a2c072dbc1fac88d3cfbc37c 3.32MB / 3.32MB 0.6s => => sha256:b7885ba641278df6e419c1c888a8981d86aeb940cc1e17c99d07acb42ddd8d57 13.74MB / 13.74MB 1.5s => => sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 1.86kB / 1.86kB 0.0s => => sha256:ab172a9f49e99303f4e3f13978812bad85a6ccb74545e95f0a3c60dcf550642c 244B / 244B 0.8s => => sha256:f5995b23a281d55dc1aada4458690373f1fe4cf0ac1d711a01a4baaf10bff0bb 3.13MB / 3.13MB 1.3s => => extracting sha256:f546e941f15b76df3d982d56985432b05bc065e3923fb35be25a4d33d5c0f911 1.5s => => extracting sha256:24935aba99a712c5a6efc86118762b0f914e1577a2c072dbc1fac88d3cfbc37c 0.1s => => extracting sha256:b7885ba641278df6e419c1c888a8981d86aeb940cc1e17c99d07acb42ddd8d57 0.4s => => extracting sha256:ab172a9f49e99303f4e3f13978812bad85a6ccb74545e95f0a3c60dcf550642c 0.0s => => extracting sha256:f5995b23a281d55dc1aada4458690373f1fe4cf0ac1d711a01a4baaf10bff0bb 0.2s => [internal] settings cache mount permissions 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.0s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.1s => [internal] load build context 0.0s => => transferring context: 72.08kB 0.0s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.0s => [scripts 1/17] COPY < [scripts 2/17] COPY < [scripts 3/17] COPY < [scripts 4/17] COPY < [scripts 5/17] COPY < [scripts 6/17] COPY < [scripts 7/17] COPY < [scripts 8/17] COPY < [scripts 9/17] COPY < [scripts 10/17] COPY < [scripts 11/17] COPY < [scripts 12/17] COPY < [scripts 13/17] COPY < [scripts 14/17] COPY < [scripts 15/17] COPY < [scripts 16/17] COPY < [scripts 17/17] COPY < [main 2/18] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.1s => [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 43.5s => [main 3/18] RUN bash /scripts/docker/install_os_dependencies.sh runtime 21.6s => [main 4/18] COPY --from=scripts common.sh /scripts/docker/ 0.0s => [main 5/18] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.0s => [main 6/18] RUN bash /scripts/docker/install_mysql.sh prod && bash /scripts/docker/install_mssql.sh prod && bash /scripts/docker 19.6s => [airflow-build-image 4/16] COPY --from=scripts common.sh /scripts/docker/ 0.0s => [airflow-build-image 5/16] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.0s => [airflow-build-image 6/16] RUN bash /scripts/docker/install_mysql.sh dev && bash /scripts/docker/install_mssql.sh dev && bash /s 15.2s => [airflow-build-image 7/16] RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password --quiet "airflow" - 0.3s => [airflow-build-image 8/16] COPY --chown=50000:0 Dockerfile /docker-context-files 0.0s => [airflow-build-image 9/16] RUN if [[ -f /docker-context-files/pip.conf ]]; then mkdir -p /home/airflow/.config/pip; cp /d 0.3s => [airflow-build-image 10/16] COPY --from=scripts common.sh install_packaging_tools.sh install_airflow_dependencies_from_branch_tip.sh 0.1s => [airflow-build-image 11/16] RUN bash /scripts/docker/install_packaging_tools.sh; bash /scripts/docker/create_prod_venv.sh; if [[ f 9.2s => [airflow-build-image 12/16] COPY --chown=airflow:0 Dockerfile /Dockerfile 0.0s => [airflow-build-image 13/16] WORKDIR /opt/airflow 0.0s => [airflow-build-image 14/16] COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh install_additional_dependenci 0.0s => ERROR [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-arm64-9,target=/tmp/.cache/pip,uid=50000 if 0.6s ------ > [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-arm64-9,target=/tmp/.cache/pip,uid=50000 if [[ false == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find "/home/airflow/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find "/home/airflow/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find "/home/airflow/.local" -executable ! -type l -print0 | xargs --null chmod g+x; find "/home/airflow/.local" ! -type l -print0 | xargs --null chmod g+rw: 0.097 0.097 Using 'pip' to install Airflow 0.097 0.117 0.117 Downloading constraints from https://raw.githubusercontent.com/apache/airflow/constraints-2.8.2/constraints-3.8.txt to /home/airflow/constraints.txt 0.117 0.242 PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/mssql-tools/bin 0.630 Installed pip: pip 24.0 from /home/airflow/.local/lib/python3.8/site-packages/pip (python 3.8): /home/airflow/.local/bin/pip 0.630 Using 'pip' to install Airflow 0.630 0.630 ERROR! You can only use --editable flag when installing airflow from sources! 0.630 Current installation method is 'apache-airflow and should be '.' ------ Dockerfile:1577 -------------------- 1576 | # hadolint ignore=SC2086, SC2010, DL3042 1577 | >>> RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \ 1578 | >>> if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \ 1579 | >>> bash /scripts/docker/install_from_docker_context_files.sh; \ 1580 | >>> fi; \ 1581 | >>> if ! airflow version 2>/dev/null >/dev/null; then \ 1582 | >>> bash /scripts/docker/install_airflow.sh; \ 1583 | >>> fi; \ 1584 | >>> if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \ 1585 | >>> bash /scripts/docker/install_additional_dependencies.sh; \ 1586 | >>> fi; \ 1587 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; \ 1588 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; \ 1589 | >>> # make sure that all directories and files in .local are also group accessible 1590 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" -executable ! -type l -print0 | xargs --null chmod g+x; \ 1591 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" ! -type l -print0 | xargs --null chmod g+rw 1592 | -------------------- ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == \"true\" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n \"${ADDITIONAL_PYTHON_DEPS}\" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local\" -executable ! -type l -print0 | xargs --null chmod g+x; find \"${AIRFLOW_USER_HOME_DIR}/.local\" ! -type l -print0 | xargs --null chmod g+rw" did not complete successfully: exit code: 1 ```
yehoshuadimarsky commented 8 months ago

Tried it now also in my GH Codespaces env, appears to be the same error.

Setup:

 ~/Documents/GitHub/airflow-upstream/ [use-uv-for-prod-image] gh codespace ssh
? Choose codespace: apache/airflow (use-uv-for-prod-image): fictional orbit
Linux 53324cd94090 6.2.0-1019-azure #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@53324cd94090:/workspaces/airflow# git status
On branch use-uv-for-prod-image
Your branch is up to date with 'origin/use-uv-for-prod-image'.

nothing to commit, working tree clean
root@53324cd94090:/workspaces/airflow# git pull
remote: Enumerating objects: 184, done.
remote: Counting objects: 100% (182/182), done.
remote: Compressing objects: 100% (126/126), done.
remote: Total 184 (delta 69), reused 153 (delta 51), pack-reused 2
Receiving objects: 100% (184/184), 340.68 KiB | 9.21 MiB/s, done.
Resolving deltas: 100% (69/69), completed with 15 local objects.
From https://github.com/apache/airflow
 + 8773cb8d8c...e54302f882 use-uv-for-prod-image          -> origin/use-uv-for-prod-image  (forced update)
   32bf0817f2..b921d4cd6f  constraints-main               -> origin/constraints-main
   a7691b787b..73a632a5a0  main                           -> origin/main
 * [new branch]            update-helm-check-instructions -> origin/update-helm-check-instructions
 * [new tag]               helm-chart/1.13.0rc1           -> helm-chart/1.13.0rc1
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
root@53324cd94090:/workspaces/airflow# git pull --rebase
Successfully rebased and updated refs/heads/use-uv-for-prod-image.
root@53324cd94090:/workspaces/airflow# git pull
Already up to date.
root@53324cd94090:/workspaces/airflow# git status
On branch use-uv-for-prod-image
Your branch is up to date with 'origin/use-uv-for-prod-image'.

nothing to commit, working tree clean
root@53324cd94090:/workspaces/airflow# git remote -v
origin  https://github.com/apache/airflow (fetch)
origin  https://github.com/apache/airflow (push)
root@53324cd94090:/workspaces/airflow# export AIRFLOW_VERSION=2.8.2
root@53324cd94090:/workspaces/airflow# export DOCKER_BUILDKIT=1

Command and error:

```bash root@53324cd94090:/workspaces/airflow# docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \ --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \ --tag "my-tag:0.0.1" [+] Building 70.2s (61/74) docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 72.05kB 0.0s => [internal] load .dockerignore 0.2s => => transferring context: 3.08kB 0.0s => resolve image config for docker.io/docker/dockerfile:1.4 0.4s => [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s => CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s => [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.3s => [auth] library/python:pull token for registry-1.docker.io 0.0s => CACHED [internal] preparing inline document 0.0s => CACHED [internal] preparing inline document 0.2s => [internal] preparing inline document 0.9s => [internal] preparing inline document 1.0s => [airflow-build-image 1/16] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631d 0.0s => [internal] preparing inline document 0.0s => [internal] preparing inline document 0.5s => [internal] preparing inline document 1.1s => [internal] preparing inline document 0.0s => [internal] preparing inline document 0.0s => CACHED [internal] settings cache mount permissions 0.0s => [internal] preparing inline document 0.0s => CACHED [internal] preparing inline document 0.0s => CACHED [internal] preparing inline document 0.0s => CACHED [internal] preparing inline document 0.0s => [internal] preparing inline document 1.1s => [internal] load build context 0.1s => => transferring context: 72.05kB 0.0s => [internal] preparing inline document 1.0s => [internal] preparing inline document 0.3s => CACHED [scripts 1/17] COPY < CACHED [scripts 2/17] COPY < CACHED [scripts 3/17] COPY < CACHED [scripts 4/17] COPY < [scripts 5/17] COPY < [scripts 6/17] COPY < [scripts 7/17] COPY < [scripts 8/17] COPY < [scripts 9/17] COPY < [scripts 10/17] COPY < [scripts 11/17] COPY < [scripts 12/17] COPY < [scripts 13/17] COPY < [scripts 14/17] COPY < [scripts 15/17] COPY < [scripts 16/17] COPY < [scripts 17/17] COPY < CACHED [airflow-build-image 2/16] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.0s => CACHED [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 0.0s => [airflow-build-image 4/16] COPY --from=scripts common.sh /scripts/docker/ 0.5s => CACHED [main 3/18] RUN bash /scripts/docker/install_os_dependencies.sh runtime 0.0s => [main 4/18] COPY --from=scripts common.sh /scripts/docker/ 0.6s => [airflow-build-image 5/16] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.5s => [main 5/18] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.6s => [airflow-build-image 6/16] RUN bash /scripts/docker/install_mysql.sh dev && bash /scripts/docker/install_mssql.sh dev && bash /s 40.7s => [main 6/18] RUN bash /scripts/docker/install_mysql.sh prod && bash /scripts/docker/install_mssql.sh prod && bash /scripts/docker 40.6s => [airflow-build-image 7/16] RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password --quiet "airflow" - 1.3s => [airflow-build-image 8/16] COPY --chown=50000:0 Dockerfile /docker-context-files 0.3s => [airflow-build-image 9/16] RUN if [[ -f /docker-context-files/pip.conf ]]; then mkdir -p /home/airflow/.config/pip; cp /d 0.6s => [airflow-build-image 10/16] COPY --from=scripts common.sh install_packaging_tools.sh install_airflow_dependencies_from_branch_tip.sh 0.5s => [airflow-build-image 11/16] RUN bash /scripts/docker/install_packaging_tools.sh; bash /scripts/docker/create_prod_venv.sh; if [[ 15.5s => [airflow-build-image 12/16] COPY --chown=airflow:0 Dockerfile /Dockerfile 0.4s => [airflow-build-image 13/16] WORKDIR /opt/airflow 0.3s => [airflow-build-image 14/16] COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh install_additional_dependenci 0.5s => ERROR [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-amd64-9,target=/tmp/.cache/pip,uid=50000 if 1.6s ------ > [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-amd64-9,target=/tmp/.cache/pip,uid=50000 if [[ false == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find "/home/airflow/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find "/home/airflow/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find "/home/airflow/.local" -executable ! -type l -print0 | xargs --null chmod g+x; find "/home/airflow/.local" ! -type l -print0 | xargs --null chmod g+rw: 0.512 0.512 Using 'pip' to install Airflow 0.512 0.546 0.546 Downloading constraints from https://raw.githubusercontent.com/apache/airflow/constraints-2.8.2/constraints-3.8.txt to /home/airflow/constraints.txt 0.546 0.803 PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/mssql-tools/bin 1.428 Installed pip: pip 24.0 from /home/airflow/.local/lib/python3.8/site-packages/pip (python 3.8): /home/airflow/.local/bin/pip 1.428 Using 'pip' to install Airflow 1.428 1.428 ERROR! You can only use --editable flag when installing airflow from sources! 1.428 Current installation method is 'apache-airflow and should be '.' ------ Dockerfile:1577 -------------------- 1576 | # hadolint ignore=SC2086, SC2010, DL3042 1577 | >>> RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \ 1578 | >>> if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \ 1579 | >>> bash /scripts/docker/install_from_docker_context_files.sh; \ 1580 | >>> fi; \ 1581 | >>> if ! airflow version 2>/dev/null >/dev/null; then \ 1582 | >>> bash /scripts/docker/install_airflow.sh; \ 1583 | >>> fi; \ 1584 | >>> if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \ 1585 | >>> bash /scripts/docker/install_additional_dependencies.sh; \ 1586 | >>> fi; \ 1587 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; \ 1588 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; \ 1589 | >>> # make sure that all directories and files in .local are also group accessible 1590 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" -executable ! -type l -print0 | xargs --null chmod g+x; \ 1591 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" ! -type l -print0 | xargs --null chmod g+rw 1592 | -------------------- ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == \"true\" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n \"${ADDITIONAL_PYTHON_DEPS}\" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local\" -executable ! -type l -print0 | xargs --null chmod g+x; find \"${AIRFLOW_USER_HOME_DIR}/.local\" ! -type l -print0 | xargs --null chmod g+rw" did not complete successfully: exit code: 1 ```
potiuk commented 8 months ago

Should be fine now. Missed a case there.

yehoshuadimarsky commented 8 months ago

That works! Thanks

potiuk commented 8 months ago

Thanks for helping with tests. Also --build-arg AIRFLOW_USE_UV=true should build the image using uv (not only allowing you to use uv to install) - which should be much faster than building image with pip.

yehoshuadimarsky commented 8 months ago

Not sure if this matters this much, but building the Docker image on Windows still fails, even after I ran git config --global core.autocrlf

``` PS C:\Users\ydima\Documents\GitHub\airflow-upstream> $env:AIRFLOW_VERSION="2.8.2" PS C:\Users\ydima\Documents\GitHub\airflow-upstream> $env:DOCKER_BUILDKIT=1 PS C:\Users\ydima\Documents\GitHub\airflow-upstream> docker build . --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" --tag "my-tag:0.0.1" [+] Building 10.4s (44/72) docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 74.34kB 0.0s => resolve image config for docker.io/docker/dockerfile:1.4 0.4s => docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 1.0s => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB 0.0s => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB 0.3s => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB 0.0s => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B 0.0s => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 0.6s => [internal] load .dockerignore 0.0s => => transferring context: 3.21kB 0.0s => [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.4s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.2s => [internal] settings cache mount permissions 0.2s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.1s => [airflow-build-image 1/16] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 7.3s => => resolve docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 0.1s => => sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 1.86kB / 1.86kB 0.0s => => sha256:3e9292a5e3bc66896a7c1dd89d59626123fbc68f9c8d45db96eacd53b0ad4580 1.37kB / 1.37kB 0.0s => => sha256:809c394c4cffdc8d84af5418b1ecf9c3195b6b9b9b3412690951f30677727a56 6.97kB / 6.97kB 0.0s => => sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56 29.12MB / 29.12MB 1.5s => => sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d 3.51MB / 3.51MB 0.4s => => sha256:07b545b886b2fba30f518b093792e37c6cd9cf02468897770202912012325b53 13.75MB / 13.75MB 1.0s => => sha256:f86bc27bff6164ce6b0a86a633bcd9640ff7467bc1f5e4eb824411ac270ed5ca 243B / 243B 0.4s => => sha256:1a56bca2cd81c555af1702d6725f31abff239026723801d1f5adf4aad7c5e14c 3.13MB / 3.13MB 0.8s => => extracting sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56 3.3s => => extracting sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d 0.4s => => extracting sha256:07b545b886b2fba30f518b093792e37c6cd9cf02468897770202912012325b53 1.1s => => extracting sha256:f86bc27bff6164ce6b0a86a633bcd9640ff7467bc1f5e4eb824411ac270ed5ca 0.0s => => extracting sha256:1a56bca2cd81c555af1702d6725f31abff239026723801d1f5adf4aad7c5e14c 0.5s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.2s => [internal] load build context 0.2s => => transferring context: 74.34kB 0.0s => [internal] preparing inline document 0.1s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.3s => [internal] preparing inline document 0.2s => [internal] preparing inline document 0.3s => [scripts 1/17] COPY < [scripts 2/17] COPY < [scripts 3/17] COPY < [scripts 4/17] COPY < [scripts 5/17] COPY < [scripts 6/17] COPY < [scripts 7/17] COPY < [scripts 8/17] COPY < [scripts 9/17] COPY < [scripts 10/17] COPY < [scripts 11/17] COPY < [scripts 12/17] COPY < [scripts 13/17] COPY < [scripts 14/17] COPY < [scripts 15/17] COPY < [scripts 16/17] COPY < [scripts 17/17] COPY < [airflow-build-image 2/16] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.4s => ERROR [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 0.6s => ERROR [main 3/18] RUN bash /scripts/docker/install_os_dependencies.sh runtime 0.6s ------ > [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev: : invalid option name/install_os_dependencies.sh: line 2: set: pipefail ------ ------ > [main 3/18] RUN bash /scripts/docker/install_os_dependencies.sh runtime: : invalid option name/install_os_dependencies.sh: line 2: set: pipefail ------ Dockerfile:1654 -------------------- 1652 | 1653 | COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 1654 | >>> RUN bash /scripts/docker/install_os_dependencies.sh runtime 1655 | 1656 | # Having the variable in final image allows to disable providers manager warnings when -------------------- ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c bash /scripts/docker/install_os_dependencies.sh runtime" did not complete successfully: exit code: 2 ```
potiuk commented 8 months ago

You likely need to check it out from scratch for autocrlf to be effective.