search

astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
351 stars 70 forks source link

Cannot add root CA cert before "astro dev start" installs dependencies #1706

Closed KarolGongola closed 1 month ago

KarolGongola commented 1 month ago

Context

We have Windows machines and they are connected to VPN with zscaler. We have some network firewall which intercepts our trafic. It means we are receiving firewall certificate instead of original website cert. So we need to add rootCA properly. As default container engine we have podman desktop.

Steps To Reproduce I have following Dockerfile:

FROM quay.io/astronomer/astro-runtime:12.0.0
USER root
COPY zscalerRootCA.cer /usr/local/share/ca-certificates/
RUN update-ca-certificates
USER astro

When I run "astro docker start" I am getting error like this (in power shell as well as in wsl):

PS XYZ> astro dev start --no-cache
[build -t astro-win_03adb2/airflow:latest -f Dockerfile . --no-cache]
STEP 1/5: FROM quay.io/astronomer/astro-runtime:12.0.0
STEP 2/12: COPY packages.txt .
time="2024-08-21T15:19:24+02:00" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> e2543bff11df
STEP 3/12: USER root
time="2024-08-21T15:19:25+02:00" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> ec0e20b73739
STEP 4/12: RUN /usr/local/bin/install-system-packages
time="2024-08-21T15:19:26+02:00" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> 19391dbacb01
STEP 5/12: COPY requirements.txt .
time="2024-08-21T15:19:27+02:00" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format"
--> b4a66e97dc20
STEP 6/12: RUN /usr/local/bin/install-python-dependencies
Looking in indexes: https://pip.astronomer.io/v2/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))': /v2/astro-run-dag/

Issue It looks like there are some additional steps like "RUN /usr/local/bin/install-python-dependencies" which are executed before my changes in Dockerfile. Should it work this way? If yes. How can I add my root CA cert before installation of dependencies?

What CLI Version did you experience this bug? Astro CLI Version: 1.28.1

This CLI bug is related to which Astronomer Platform?

What Operating System is the above CLI installed on? Windows - Powers Shell and WSL

KarolGongola commented 1 month ago

Closing it, because it can be done with base image: https://www.astronomer.io/docs/astro/runtime-image-architecture#base-images So when we use base image, we can add cert and then install packages and requirements.