search

aws / aws-mwaa-local-runner

This repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.
MIT No Attribution
672 stars 683 forks source link

Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' #374

Open hlherrera opened 5 months ago

hlherrera commented 5 months ago

Version 2.8.1 Error aws-mwaa-local-runner-2_8-local-runner-1 | [2024-04-22T00:05:58.985+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'

requirements.txt file:

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"

apache-airflow[cncf.kubernetes]==2.8.1
apache-airflow-providers-amazon==8.16.0
kubernetes==23.6.0
kubernetes-asyncio==24.2.3
cachetools==5.3.2

Command: ./mwaa-local-env start

Logs:

aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: annotated-types>=0.4.0 in ./.local/lib/python3.11/site-packages (from pydantic>=2.3.0->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (0.6.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: pydantic-core==2.14.6 in ./.local/lib/python3.11/site-packages (from pydantic>=2.3.0->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.14.6)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: docutils in ./.local/lib/python3.11/site-packages (from python-daemon>=3.0.0->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (0.20.1)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: text-unidecode>=1.3 in ./.local/lib/python3.11/site-packages (from python-slugify>=5.0->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.3)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: scramp<1.5.0,>=1.2.0 in ./.local/lib/python3.11/site-packages (from redshift_connector>=2.0.888->apache-airflow-providers-amazon==8.16.0->-r /usr/local/airflow/requirements/requirements.txt (line 4)) (1.4.4)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: beautifulsoup4<5.0.0,>=4.7.0 in ./.local/lib/python3.11/site-packages (from redshift_connector>=2.0.888->apache-airflow-providers-amazon==8.16.0->-r /usr/local/airflow/requirements/requirements.txt (line 4)) (4.12.2)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: lxml>=4.6.5 in ./.local/lib/python3.11/site-packages (from redshift_connector>=2.0.888->apache-airflow-providers-amazon==8.16.0->-r /usr/local/airflow/requirements/requirements.txt (line 4)) (5.1.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: charset-normalizer<4,>=2 in ./.local/lib/python3.11/site-packages (from requests->kubernetes==23.6.0->-r /usr/local/airflow/requirements/requirements.txt (line 5)) (3.3.2)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.11/site-packages (from requests->kubernetes==23.6.0->-r /usr/local/airflow/requirements/requirements.txt (line 5)) (3.6)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: greenlet!=0.4.17 in ./.local/lib/python3.11/site-packages (from sqlalchemy<2.0,>=1.4.28->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (3.0.3)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: requests_toolbelt in ./.local/lib/python3.11/site-packages (from apache-airflow-providers-http->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.0.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in ./.local/lib/python3.11/site-packages (from httpx->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (0.16.3)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in ./.local/lib/python3.11/site-packages (from httpx->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.5.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: sniffio in ./.local/lib/python3.11/site-packages (from httpx->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.3.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc==1.22.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.22.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: opentelemetry-exporter-otlp-proto-http==1.22.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.22.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: backoff<3.0.0,>=1.10.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.2.1)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: googleapis-common-protos~=1.52 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.62.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: grpcio<2.0.0,>=1.0.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.60.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.22.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.22.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: opentelemetry-proto==1.22.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.22.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: opentelemetry-sdk~=1.22.0 in ./.local/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.22.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: protobuf<5.0,>=3.19 in ./.local/lib/python3.11/site-packages (from opentelemetry-proto==1.22.0->opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (4.25.2)
aws-mwaa-local-runner-2_8-local-runner-1  | Collecting oauthlib>=3.0.0
aws-mwaa-local-runner-2_8-local-runner-1  |   Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
aws-mwaa-local-runner-2_8-local-runner-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 26.9 MB/s eta 0:00:00
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: soupsieve>1.2 in ./.local/lib/python3.11/site-packages (from beautifulsoup4<5.0.0,>=4.7.0->redshift_connector>=2.0.888->apache-airflow-providers-amazon==8.16.0->-r /usr/local/airflow/requirements/requirements.txt (line 4)) (2.5)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: pycparser in ./.local/lib/python3.11/site-packages (from cffi>=1.12->cryptography>=0.9.3->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.21)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: dnspython>=1.15.0 in ./.local/lib/python3.11/site-packages (from email-validator<2,>=1.0.5->flask-appbuilder==4.3.10->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.4.2)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: Babel>=2.3 in ./.local/lib/python3.11/site-packages (from Flask-Babel<3,>=1->flask-appbuilder==4.3.10->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.14.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: limits>=2.8 in ./.local/lib/python3.11/site-packages (from Flask-Limiter<4,>3->flask-appbuilder==4.3.10->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (3.7.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: ordered-set<5,>4 in ./.local/lib/python3.11/site-packages (from Flask-Limiter<4,>3->flask-appbuilder==4.3.10->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (4.1.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: h11<0.15,>=0.13 in ./.local/lib/python3.11/site-packages (from httpcore<0.17.0,>=0.15.0->httpx->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (0.14.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: anyio<5.0,>=3.0 in ./.local/lib/python3.11/site-packages (from httpcore<0.17.0,>=0.15.0->httpx->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (4.2.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.11/site-packages (from importlib-metadata<7.0,>=6.0->opentelemetry-api>=1.15.0->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (3.17.0)
aws-mwaa-local-runner-2_8-local-runner-1  | Collecting pyasn1<0.6.0,>=0.4.6
aws-mwaa-local-runner-2_8-local-runner-1  |   Downloading pyasn1-0.5.1-py2.py3-none-any.whl (84 kB)
aws-mwaa-local-runner-2_8-local-runner-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.9/84.9 kB 10.1 MB/s eta 0:00:00
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: asn1crypto>=1.5.1 in ./.local/lib/python3.11/site-packages (from scramp<1.5.0,>=1.2.0->redshift_connector>=2.0.888->apache-airflow-providers-amazon==8.16.0->-r /usr/local/airflow/requirements/requirements.txt (line 4)) (1.5.1)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: importlib-resources>=1.3 in ./.local/lib/python3.11/site-packages (from limits>=2.8->Flask-Limiter<4,>3->flask-appbuilder==4.3.10->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (6.1.1)
aws-mwaa-local-runner-2_8-local-runner-1  | Requirement already satisfied: opentelemetry-semantic-conventions==0.43b0 in ./.local/lib/python3.11/site-packages (from opentelemetry-sdk~=1.22.0->opentelemetry-exporter-otlp-proto-grpc==1.22.0->opentelemetry-exporter-otlp->apache-airflow[cncf.kubernetes]==2.8.1->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (0.43b0)
aws-mwaa-local-runner-2_8-local-runner-1  | Installing collected packages: websocket-client, pyasn1, oauthlib, cachetools, rsa, requests-oauthlib, pyasn1-modules, kubernetes-asyncio, google-auth, kubernetes
aws-mwaa-local-runner-2_8-local-runner-1  | Successfully installed cachetools-5.3.2 google-auth-2.26.2 kubernetes-23.6.0 kubernetes-asyncio-24.2.3 oauthlib-3.2.2 pyasn1-0.5.1 pyasn1-modules-0.3.0 requests-oauthlib-1.3.1 rsa-4.9 websocket-client-1.7.0
aws-mwaa-local-runner-2_8-local-runner-1  | 
aws-mwaa-local-runner-2_8-local-runner-1  | [notice] A new release of pip available: 22.3.1 -> 24.0
aws-mwaa-local-runner-2_8-local-runner-1  | [notice] To update, run: pip install --upgrade pip

Inside the runtime container:

aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:02:23.092+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:02:52.721+0000] {{triggerer_job_runner.py:481}} INFO - 0 triggers currently running
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:02:53.658+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:03:24.282+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:03:52.931+0000] {{triggerer_job_runner.py:481}} INFO - 0 triggers currently running
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:03:55.105+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:04:26.153+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:04:53.206+0000] {{triggerer_job_runner.py:481}} INFO - 0 triggers currently running
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:04:57.234+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:05:27.988+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:05:53.433+0000] {{triggerer_job_runner.py:481}} INFO - 0 triggers currently running
aws-mwaa-local-runner-2_8-local-runner-1  | [2024-04-22T00:05:58.985+0000] {{processor.py:376}} WARNING - Error when trying to pre-import module 'airflow.providers.cncf.kubernetes.operators.pod' found in /usr/local/airflow/dags/jupyter_notebook_poc.py: No module named 'airflow.providers.cncf'

Trying to load this dag:

from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

k = KubernetesPodOperator(
    name="hello-dry-run",
    image="debian",
    cmds=["bash", "-cx"],
    arguments=["echo", "10"],
    labels={"foo": "bar"},
    task_id="dry_run_demo",
    do_xcom_push=True,
)

k.dry_run()