search

Data472-Individual-Project-Pipeline / Data-collection-service

0 stars 0 forks source link

Module import error in airflow webserver #17

Closed Jake1441 closed 5 months ago

Jake1441 commented 5 months ago

not sure if its something to do with sequentialExecutor or localexecutors

Python 3.11.7 venv Package Version

aiohttp 3.9.5 aiosignal 1.3.1 alembic 1.13.1 anyio 4.3.0 apache-airflow 2.9.1 apache-airflow-providers-common-io 1.3.1 apache-airflow-providers-common-sql 1.13.0 apache-airflow-providers-fab 1.1.0 apache-airflow-providers-ftp 3.9.0 apache-airflow-providers-http 4.11.0 apache-airflow-providers-imap 3.6.0 apache-airflow-providers-postgres 5.11.0 apache-airflow-providers-smtp 1.7.0 apache-airflow-providers-sqlite 3.8.0 apispec 6.6.1 argcomplete 3.3.0 asgiref 3.8.1 attrs 23.2.0 Babel 2.15.0 blinker 1.8.2 cachelib 0.9.0 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 clickclick 20.10.2 colorama 0.4.6 colorlog 4.8.0 ConfigUpdater 3.2 connexion 2.14.2 cron-descriptor 1.4.3 croniter 2.0.5 cryptography 42.0.7 demoji 1.1.0 Deprecated 1.2.14 dill 0.3.8 dnspython 2.6.1 docutils 0.21.2 email_validator 2.1.1 Flask 2.2.5 Flask-AppBuilder 4.4.1 Flask-Babel 2.0.0 Flask-Caching 2.3.0 Flask-JWT-Extended 4.6.0 Flask-Limiter 3.7.0 Flask-Login 0.6.3 Flask-Session 0.5.0 Flask-SQLAlchemy 2.5.1 Flask-WTF 1.2.1 frozenlist 1.4.1 fsspec 2024.5.0 google-re2 1.1.20240501 googleapis-common-protos 1.63.0 graphviz 0.20.3 greenlet 3.0.3 grpcio 1.64.0 gunicorn 22.0.0 h11 0.14.0 httpcore 1.0.5 httpx 0.27.0 idna 3.7 importlib-metadata 7.0.0 importlib_resources 6.4.0 inflection 0.5.1 itsdangerous 2.2.0 Jinja2 3.1.4 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 lazy-object-proxy 1.10.0 limits 3.12.0 linkify-it-py 2.0.3 lockfile 0.12.2 Mako 1.3.5 markdown-it-py 3.0.0 MarkupSafe 2.1.5 marshmallow 3.21.2 marshmallow-oneofschema 3.1.1 marshmallow-sqlalchemy 0.28.2 mdit-py-plugins 0.4.1 mdurl 0.1.2 methodtools 0.4.7 more-itertools 10.2.0 multidict 6.0.5 opentelemetry-api 1.24.0 opentelemetry-exporter-otlp 1.24.0 opentelemetry-exporter-otlp-proto-common 1.24.0 opentelemetry-exporter-otlp-proto-grpc 1.24.0 opentelemetry-exporter-otlp-proto-http 1.24.0 opentelemetry-proto 1.24.0 opentelemetry-sdk 1.24.0 opentelemetry-semantic-conventions 0.45b0 ordered-set 4.1.0 packaging 24.0 pathspec 0.12.1 pendulum 3.0.0 pip 24.0 pluggy 1.5.0 prison 0.2.1 processors 0.0.7 protobuf 4.25.3 psutil 5.9.8 psycopg2-binary 2.9.9 pycparser 2.22 Pygments 2.18.0 PyJWT 2.8.0 python-daemon 3.0.1 python-dateutil 2.9.0.post0 python-nvd3 0.16.0 python-slugify 8.0.4 pytz 2024.1 PyYAML 6.0.1 referencing 0.35.1 requests 2.32.2 requests-toolbelt 1.0.0 rfc3339-validator 0.1.4 rich 13.7.1 rich-argparse 1.4.0 rpds-py 0.18.1 setproctitle 1.3.3 setuptools 65.5.0 six 1.16.0 sniffio 1.3.1 SQLAlchemy 1.4.52 SQLAlchemy-JSONField 1.0.2 SQLAlchemy-Utils 0.41.2 sqlparse 0.5.0 tabulate 0.9.0 tenacity 8.3.0 termcolor 2.4.0 text-unidecode 1.3 time-machine 2.14.1 typing_extensions 4.11.0 tzdata 2024.1 uc-micro-py 1.0.3 unicodecsv 0.14.1 universal_pathlib 0.2.2 urllib3 2.2.1 Werkzeug 2.2.3 wirerope 0.4.7 wrapt 1.16.0 WTForms 3.1.2 yarl 1.9.4 zipp 3.18.2

cfg dags_folder = /home/ubuntu/Data-collection-service/dags

AIRFLOW_HOME = /home/ubuntu/Data-collection-service

directory listing tree dags/ dags/ ├── pycache │   ├── dus15.cpython-311.pyc │   ├── hpa117.cpython-311.pyc │   ├── hra80.cpython-311.pyc │   ├── hwa205.cpython-311.pyc │   ├── jjm148.cpython-311.pyc │   ├── owners_create.cpython-311.pyc │   ├── owners_update.cpython-311.pyc │   ├── rna104.cpython-311.pyc │   ├── ruben.cpython-311.pyc │   ├── sss135.cpython-311.pyc │   └── tya51.cpython-311.pyc ├── dus15.py ├── hpa117.py ├── hra80.py ├── hwa205.py ├── jjm148.py ├── owners_create.py ├── owners_update.py ├── processors │   ├── init.py │   ├── pycache │   │   └── owner_processor.cpython-311.pyc │   ├── dus15_processor.py │   ├── hpa117_processor.py │   ├── hra80_processor.py │   ├── hwa205_processor.py │   ├── jjm148_processor.py │   ├── owner_processor.py │   ├── rna104_processor.py │   ├── ruben_processor.py │   ├── sss135_processor.py │   └── tya51_processor.py ├── rna104.py ├── ruben.py ├── sss135.py

dag errors [ { "filepath": "/home/ubuntu/Data-collection-service/dags/rna104.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/rna104.py\", line 7, in \n from processors.rna104_processor import Rna104Processor\nModuleNotFoundError: No module named 'processors.rna104_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/hpa117.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/hpa117.py\", line 7, in \n from processors.hpa117_processor import Hpa117Processor\nModuleNotFoundError: No module named 'processors.hpa117_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/hwa205.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/hwa205.py\", line 7, in \n from processors.hwa205_processor import Hwa205Processor\nModuleNotFoundError: No module named 'processors.hwa205_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/ruben.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/ruben.py\", line 7, in \n from processors.ruben_processor import RubenProcessor\nModuleNotFoundError: No module named 'processors.ruben_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/sss135.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/sss135.py\", line 7, in \n from processors.sss135_processor import Sss135Processor\nModuleNotFoundError: No module named 'processors.sss135_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/tya51.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/tya51.py\", line 7, in \n from processors.tya51_processor import Tya51Processor\nModuleNotFoundError: No module named 'processors.tya51_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/owners_create.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/owners_create.py\", line 11, in \n from processors.owner_processor import OwnerManager\nModuleNotFoundError: No module named 'processors.owner_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/jjm148.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/jjm148.py\", line 7, in \n from processors.jjm148_processor import Jjm148Processor\nModuleNotFoundError: No module named 'processors.jjm148_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/dus15.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/dus15.py\", line 7, in \n from processors.dus15_processor import Dus15Processor\nModuleNotFoundError: No module named 'processors.dus15_processor'\n" }, { "filepath": "/home/ubuntu/Data-collection-service/dags/hra80.py", "error": "Traceback (most recent call last):\n File \"\", line 241, in _call_with_frames_removed\n File \"/home/ubuntu/Data-collection-service/dags/hra80.py\", line 7, in \n from processors.hra80_processor import Hra80Processor\nModuleNotFoundError: No module named 'processors.hra80_processor'\n" } ]

Network tcp 0 0 0.0.0.0:8000 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:8080 0.0.0.0: LISTEN 4271/gunicorn: mast

docker CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES d2673b229f8d postgres "docker-entrypoint.s…" About an hour ago Up 20 minutes 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp

processses ps -ef | grep airflow sil 4202 1119 3 10:47 pts/2 00:00:41 /home/ubuntu/Data-collection-service/.venv/bin/python3 /home/ubuntu/Data-collection-service/.venv/bin/airflow scheduler sil 4203 1119 0 10:47 pts/2 00:00:10 /home/ubuntu/Data-collection-service/.venv/bin/python3 /home/ubuntu/Data-collection-service/.venv/bin/airflow webserver -p 8080 sil 4271 4203 0 10:47 pts/2 00:00:03 gunicorn: master [airflow-webserver] sil 4274 4202 1 10:47 pts/2 00:00:23 airflow scheduler -- DagFileProcessorManager sil 4302 4271 0 10:47 pts/2 00:00:02 [ready] gunicorn: worker [airflow-webserver] sil 4303 4271 0 10:47 pts/2 00:00:01 [ready] gunicorn: worker [airflow-webserver] sil 4304 4271 0 10:47 pts/2 00:00:02 [ready] gunicorn: worker [airflow-webserver] sil 4305 4271 0 10:47 pts/2 00:00:02 [ready] gunicorn: worker [airflow-webserver] sil 12834 1119 0 11:10 pts/2 00:00:00 grep --color=auto airflow

aemooooon commented 5 months ago

@Jake1441 I used local executors, but I do not know why.