Open daniel-bartley opened 7 months ago
nbonnotte
commented
7 months ago It looks like the container composer-local-dev-tobe stopped unexpectedly. Do you see anything useful in its logs? You can get them with
docker container logs composer-local-dev-tobe
daniel-bartley
commented
7 months ago $ docker container logs composer-local-dev-tobe
ENV=
+ echo ENV=
+ main
+ sudo chown airflow:airflow airflow
+ [ = True ]
+ run_airflow_as_airflow_user
+ echo Running Airflow as user airflow(999)
Running Airflow as user airflow(999)
+ sudo cat
+ sudo chmod +x /usr/local/bin/run
+ init_airflow
+ run mkdir -p /home/airflow/gcs/dags
+ run mkdir -p /home/airflow/gcs/plugins
+ run mkdir -p
mkdir: missing operand
Try 'mkdir --help' for more information.It seems the variable AIRFLOW__CORE__DATA_FOLDER is not set, and it causes this line to fail. By default, it should be "/home/airflow/gcs/data", so I'm a bit surprised there is no value :thinking:
daniel-bartley
commented
7 months ago I used pipx install -e composer-local-dev to install it
doubled checked:
~/composer-local-dev (debug-composer-dev)
$ AIRFLOW__CORE__DATA_FOLDER="/home/airflow/gcs/data" composer-dev start foo --debug
2024-02-14T12:02:00 INFO composer_local_dev.files:Found files.py:42
following local environments:
C:\Users\dbrtly\
composer-local-dev\composer\foo
INFO composer_local_dev.files:Searching for files.py:54
provided local environment name: foo
INFO composer_local_dev.environment: environment.py:367
Loading configuration file from
C:\Users\dbrtly\
composer-local-dev\composer\foo
\config.json
INFO composer_local_dev.environment: environment.py:120
Loading environment variables
from
C:\Users\dbrtly\
composer-local-dev\composer\foo
\variables.env
DEBUG docker.utils.config:Trying paths: config.py:21
['C:\\Users\\dbrtly\\.docker\\config
.json',
'C:\\Users\\dbrtly\\.dockercfg']
DEBUG docker.utils.config:Found file at config.py:25
path:
C:\Users\dbrtly\.docker\config.json
DEBUG docker.auth:Found 'credsStore' section auth.py:187
DEBUG urllib3.connectionpool:http: connectionpool.py:549
//localhost:None "GET
/version HTTP/1.1" 200 None
Starting foo composer environment...
INFO composer_local_dev.environment: environment.py:227
Asserting that
composer-2.5.3-airflow-2-6-3
composer image version exists
DEBUG google.auth._default:Checking None _default.py:255
for explicit credentials as part
of auth process...
DEBUG google.auth._default:Checking _default.py:228
Cloud SDK credentials as part of
auth process...
2024-02-14T12:02:02 DEBUG composer_local_dev.environment: environment.py:233
GetTagRequest for
composer-2.5.3-airflow-2-6-3:
name:
"projects/cloud-airflow-release
r/locations/us/repositories/air
flow-worker-scheduler-2-6-3/pac
kages/airflow-worker-scheduler-
2-6-3/tags/composer-2.5.3-airfl
ow-2-6-3"
DEBUG google.auth.transport.requests:Mak requests.py:185
ing request: POST
https://oauth2.googleapis.com/toke
n
DEBUG urllib3.connectionpool:Star connectionpool.py:1055
ting new HTTPS connection
(1):
oauth2.googleapis.com:443
DEBUG urllib3.connectionpool:https connectionpool.py:549
://oauth2.googleapis.com:443
"POST /token HTTP/1.1" 200
None
2024-02-14T12:02:03 DEBUG urllib3.connectionpool:http: connectionpool.py:549
//localhost:None "GET
/v1.43/containers/composer-l
ocal-dev-foo/json HTTP/1.1"
200 None
2024-02-14T12:02:04 DEBUG urllib3.connectionpool:http: connectionpool.py:549
//localhost:None "POST
/v1.43/containers/13b7462c6e
4de5d28dc3725e0bf4b0cc63d82e
5f278117ac364e3d92d4119d4a/s
tart HTTP/1.1" 204 0
DEBUG urllib3.connectionpool:http: connectionpool.py:549
//localhost:None "GET
/v1.43/containers/composer-l
ocal-dev-foo/json HTTP/1.1"
200 None
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Environment failed to start. │
│ │
│ To print debug messages please add --debug flag. │
╰──────────────────────────────────────────────────────────────────────────────╯
The environment variables can be overridden in composer/{yourEnv}/variables.env. What happens if you set AIRFLOW__CORE__DATA_FOLDER there?
daniel-bartley
commented
7 months ago progress
is "composer_requirements.txt" meant to be included in the repository?
$ composer-dev restart foo
2024-02-15T01:01:51.483538673Z + echo ENV=
2024-02-15T01:01:51.483749773Z + main
2024-02-15T01:01:51.483768373Z + sudo chown airflow:airflow airflow
2024-02-15T01:01:51.483626073Z ENV=
2024-02-15T01:01:51.615417871Z + [ = True ]
2024-02-15T01:01:51.615506871Z + run_airflow_as_airflow_user
2024-02-15T01:01:51.615525271Z + echo Running Airflow as user airflow(999)
2024-02-15T01:01:51.615432371Z Running Airflow as user airflow(999)
2024-02-15T01:01:51.619766171Z + sudo cat
2024-02-15T01:01:51.662674671Z + run mkdir -p /home/airflow/gcs/dags
2024-02-15T01:01:51.673475171Z + run mkdir -p /home/airflow/gcs/plugins
2024-02-15T01:01:51.678367370Z + run mkdir -p "/home/airflow/gcs/data"
2024-02-15T01:01:51.686361570Z + [ -f /var/local/setup_python_command.sh ]
2024-02-15T01:01:51.690279070Z + run pip3 install --upgrade -r composer_requirements.txt
2024-02-15T01:01:54.035113438Z Defaulting to user installation because normal site-packages
is not writeable
2024-02-15T01:01:55.486477737Z WARNING: Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))': /simple/boto3/
2024-02-15T01:01:56.125469151Z WARNING: Retrying (Retry(total=3, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))': /simple/boto3/
2024-02-15T01:01:57.234312077Z WARNING: Retrying (Retry(total=2, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))': /simple/boto3/
2024-02-15T01:01:59.363699626Z WARNING: Retrying (Retry(total=1, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))': /simple/boto3/
2024-02-15T01:02:03.485662338Z WARNING: Retrying (Retry(total=0, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))': /simple/boto3/
2024-02-15T01:02:03.576133716Z Could not fetch URL https://pypi.org/simple/boto3/: There wasa problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443):
Max retries exceeded with url: /simple/boto3/ (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))) - skipping
2024-02-15T01:02:03.581660320Z ERROR: Could not find a version that satisfies the
requirement boto3 (from -r composer_requirements.txt (line 1)) (from versions: none)
2024-02-15T01:02:03.581851620Z ERROR: No matching distribution found for boto3 (from -r
composer_requirements.txt (line 1))
╭─ Error ──────────────────────────────────────────────────────────────────────────────────╮│ Environment failed to start. ││ ││ To print debug messages please add --debug flag. │╰──────────────────────────────────────────────────────────────────────────────────────────╯composer/foo/requirements.txt specifies no version tags for most of the libraries.
I hand-edited to add the most recent version for each as per PyPI.
2024-02-15T01:19:51.668723956Z ERROR: Could not find a version that satisfies the
requirement boto3==1.34.42 (from -r composer_requirements.txt (line 1)) (from versions:
none)
2024-02-15T01:19:51.668944556Z ERROR: No matching distribution found for boto3==1.34.42
(from -r composer_requirements.txt (line 1))The log message just before is interesting:
2024-02-15T01:02:03.576133716Z Could not fetch URL https://pypi.org/simple/boto3/: There wasa problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443):
Max retries exceeded with url: /simple/boto3/ (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed: unable to get local issuer certificate (_ssl.c:1131)'))) - skipping
2024-02-15T01:02:03.581660320Z ERROR: Could not find a version that satisfies the
requirement boto3 (from -r composer_requirements.txt (line 1)) (from versions: none)It seems pip can't even establish a proper HTTPS connection :thinking:
daniel-bartley
commented
7 months ago The log message just before is interesting:
2024-02-15T01:02:03.576133716Z Could not fetch URL https://pypi.org/simple/boto3/: There wasa problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/boto3/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))) - skipping 2024-02-15T01:02:03.581660320Z ERROR: Could not find a version that satisfies the requirement boto3 (from -r composer_requirements.txt (line 1)) (from versions: none)It seems
pipcan't even establish a proper HTTPS connection 🤔
edited for brevity:
composer/foo/config.json. Is that intended to be included with the create environment command like "path_to_project"?Nicholas, How can I help debug these?
nbonnotte
commented
7 months ago I'm sorry @daniel-bartley, I'm afraid I can't help you much more than that. I'm not even using Windows :sweat:
But I'm not even sure your issues are coming from Composer Local Dev. If you start a plain Python docker image, can you install libraries from PyPi?
daniel-bartley
commented
7 months ago Editing these lines resolves the SSL problems for me:
RUN python -m pip3 install --upgrade --no-cache-dir --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host pypi.org -r composer_requirements.txt
RUN python -m pip3 checkThe team at Astronomer say pip install --upgrade certifi is equivalent.
https://pypi.org/project/certifi/
I forked and applied dependency bumps from #31. Then successfully created an environment from a source environment from GCP. But I can't start that local environment: