search

astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
363 stars 71 forks source link

`astro dev start` results in CERTIFICATE_VERIFY_FAILED #1177

Open tdhopper opened 1 year ago

tdhopper commented 1 year ago

Describe the bug

I'm trying to run astro dev start on a Mac and get a certificate error. 

 => ERROR [5/1] RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'a  25.2s
------
 > [5/1] RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pip install --no-cache-dir --root-user-action=ignore -r requirements.txt:
#9 0.267 + grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt
#9 0.278 + pip install --no-cache-dir --root-user-action=ignore -r requirements.txt
#9 1.046 Looking in indexes: https://pip.astronomer.io/v2/
#9 1.046 Looking in links: https://pip.astronomer.io/simple/astronomer-fab-security-manager/, https://pip.astronomer.io/simple/astronomer-airflow-version-check/
#9 1.121 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-fab-security-manager/
#9 1.692 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-fab-security-manager/
#9 2.764 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-fab-security-manager/
#9 4.839 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-fab-security-manager/
#9 8.913 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-fab-security-manager/
#9 8.983 Could not fetch URL https://pip.astronomer.io/simple/astronomer-fab-security-manager/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pip.astronomer.io', port=443): Max retries exceeded with url: /simple/astronomer-fab-security-manager/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
#9 9.031 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-airflow-version-check/
#9 9.586 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-airflow-version-check/
#9 10.66 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-airflow-version-check/
#9 12.74 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-airflow-version-check/
#9 16.83 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /simple/astronomer-airflow-version-check/
#9 16.90 Could not fetch URL https://pip.astronomer.io/simple/astronomer-airflow-version-check/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pip.astronomer.io', port=443): Max retries exceeded with url: /simple/astronomer-airflow-version-check/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
#9 16.97 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /v2/astro-run-dag/
#9 17.52 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /v2/astro-run-dag/
#9 18.62 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /v2/astro-run-dag/
#9 20.68 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /v2/astro-run-dag/
#9 24.78 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))': /v2/astro-run-dag/
#9 24.87 Could not fetch URL https://pip.astronomer.io/v2/astro-run-dag/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pip.astronomer.io', port=443): Max retries exceeded with url: /v2/astro-run-dag/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
#9 24.88 ERROR: Could not find a version that satisfies the requirement astro-run-dag (from versions: none)
#9 24.88 ERROR: No matching distribution found for astro-run-dag
#9 24.95 Could not fetch URL https://pip.astronomer.io/simple/astronomer-fab-security-manager/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pip.astronomer.io', port=443): Max retries exceeded with url: /simple/astronomer-fab-security-manager/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
#9 25.01 Could not fetch URL https://pip.astronomer.io/simple/astronomer-airflow-version-check/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pip.astronomer.io', port=443): Max retries exceeded with url: /simple/astronomer-airflow-version-check/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
#9 25.09 Could not fetch URL https://pip.astronomer.io/v2/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pip.astronomer.io', port=443): Max retries exceeded with url: /v2/pip/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
------
executor failed running [/bin/bash -o pipefail -e -u -x -c if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pip install --no-cache-dir --root-user-action=ignore -r requirements.txt]: exit code: 1
Error: command 'docker build -t astro_2c01d3/airflow:latest failed: failed to execute cmd: exit status 1

What CLI Version did you experience this bug?

Astro CLI Version: 1.14.1

This CLI bug is related to which Astronomer Platform?

What Operating System is the above CLI installed on?

MacOS 13.3.1

🪜 Steps To Reproduce

brew install astro && astro dev start

tdhopper commented 1 year ago

I'm behind a that corporate VPN, so that could be the issue. However, I can access https://pip.astronomer.io/simple/ and the other urls from my browser without issue.

sunkickr commented 1 year ago

In the past users behind a corporate VPN sometimes have to add the urls as "trusted hosts" in a docker_congif/pip.conf file.

https://stackoverflow.com/questions/59287824/specifying-multiple-trusted-hosts-in-pip-conf

radonys commented 1 year ago

@tdhopper Did you find a fix?

tdhopper commented 1 year ago

no

jtrells commented 8 months ago

I had a similar issue and solved it by doing the following:

  1. Extend the Dockefile in your project based on these instructions: https://docs.astronomer.io/astro/cli/customize-dockerfile#add-a-ca-certificate-to-an-astro-runtime-image
  2. Build a custom docker image (docker build -t myimage:0.1 -f Dockerfile .)
  3. Run astro dev start with the parameter -i and pass the name of your custom docker image

Another option is to add pip.astronomer.io as a trusted-host and pass it to the image:

[global]
extra-index-url = 
trusted-host =  pypi.org
                pypi.python.org
                pip.astronomer.io
FROM quay.io/astronomer/astro-runtime:10.5.0
COPY pip.conf /home/astro/.pip/pip.conf

Build a custom image as detailed above and use it with -i in the astro dev start command

gnirs commented 6 months ago

@jtrells , i followed this and able to create a build when there are no requirement. But when i add say astronomer-cosmos into requirements, the docker build is failing due to No matching distribution found for astronomer-cosmos.

any suggestions? thank you

peterampazzo commented 5 months ago

I encountered a similar issue and found that the suggestion by @jtrells resolved the problem for me. To provide some context, the Dockerfile I used is as follows:

FROM quay.io/astronomer/astro-runtime:11.6.0

COPY pip.conf /home/astro/.pip/pip.conf

USER root
COPY <company>.crt /usr/local/share/ca-certificates/<company>/
RUN update-ca-certificates
USER astro

The pip.conf file contains the following content:

[global]
extra-index-url = 
trusted-host = 
    pypi.org
    pypi.python.org
    pip.astronomer.io

After creating a custom Docker image using this Dockerfile, I ran it using Astro CLI:

docker build -t myimage:0.1 -f Dockerfile .
astro dev start --image-name myimage:0.1

However, during the execution, I encountered the following known issue, which is documented here:

➜  dagbag git:(main) ✗ astro dev start --image-name myimage:0.1
Env file ".env" found. Loading...
[+] Running 0/0
 ⠿ postgres Error                                                                      0.0s
Error: error building, (re)creating or starting project containers: Error response from daemon: Please run 'docker login'

The solution that worked for me was to pull the Postgres image using the following command:

docker pull postgres:12.6
mulcahyk commented 2 months ago

I am have this same problem and none of the above solutions are working for me. I'm running on a Mac with a brew install astro instance. The version is 1.29.0.
My dockerfile FROM is: FROM quay.io/astronomer/astro-runtime:12.1.0 It looks like the docker build process thatastro dev start uses isn't working properly. /usr/local/bin/install-python-dependencies is being run before either the certificates are updated or the pip.conf is loaded.

My astro dev start docker logs: `[internal] load build definition from Dockerfile transferring 699/0 0.004 [internal] load metadata for quay.io/astronomer/astro-runtime:12.1.0 [internal] load .dockerignore [internal] load build context transferring 652772/0 0.021 [1/7] FROM quay.io/astronomer/astro-runtime:12.1.0@sha256:77d85a8b6cd3bdfbcb25cbdb6ffbdb4ac2c1bb390b5951b836451dbdefb6c325 [2/7] COPY packages.txt . [3/7] RUN /usr/local/bin/install-system-packages

siddiqkaithodu commented 4 days ago

Please Prioritize this guyz 👍🏻