search

Team1-TU-tech / airflow

0 stars 0 forks source link

User site-packages are not visible in this virtualenv #2

Open hamsunwoo opened 4 days ago

hamsunwoo commented 4 days ago

관련 이슈

dMario24 commented 4 days ago

해당 이슈 답변 완료

LGTM

hamsunwoo commented 4 days ago

참고 링크 https://airflow.apache.org/docs/apache-airflow/2.10.3/howto/docker-compose/index.html

hamsunwoo commented 4 days ago

실행 시 컨테이너 안에 깃이 없어 생기는 오류

320ff9130a0d
*** Found local files:
***   * /opt/airflow/logs/dag_id=yes24_to_MongoDB/run_id=manual__2024-11-21T05:25:26.440992+00:00/task_id=extract/attempt=1.log
[2024-11-21, 05:25:31 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2024-11-21, 05:25:33 UTC] {baseoperator.py:416} WARNING - PythonVirtualenvOperator.execute cannot be called outside TaskInstance!
[2024-11-21, 05:25:33 UTC] {process_utils.py:186} INFO - Executing cmd: /home/***/.local/bin/python -m virtualenv /tmp/venvj6ne93ah --python=python
[2024-11-21, 05:25:33 UTC] {process_utils.py:190} INFO - Output:
[2024-11-21, 05:25:35 UTC] {process_utils.py:194} INFO - created virtual environment CPython3.12.7.final.0-64 in 896ms
[2024-11-21, 05:25:35 UTC] {process_utils.py:194} INFO -   creator CPython3Posix(dest=/tmp/venvj6ne93ah, clear=False, no_vcs_ignore=False, global=False)
[2024-11-21, 05:25:35 UTC] {process_utils.py:194} INFO -   seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/home/***/.local/share/virtualenv)
[2024-11-21, 05:25:35 UTC] {process_utils.py:194} INFO -     added seed packages: pip==24.3.1
[2024-11-21, 05:25:35 UTC] {process_utils.py:194} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2024-11-21, 05:25:35 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venvj6ne93ah/bin/pip install -r /tmp/venvj6ne93ah/requirements.txt
[2024-11-21, 05:25:35 UTC] {process_utils.py:190} INFO - Output:
[2024-11-21, 05:25:37 UTC] {process_utils.py:194} INFO - Collecting git+https://github.com/Team1-TU-tech/crawling.git@0.3.2/dev/yes24 (from -r /tmp/venvj6ne93ah/requirements.txt (line 1))
[2024-11-21, 05:25:37 UTC] {process_utils.py:194} INFO -   Cloning https://github.com/Team1-TU-tech/crawling.git (to revision 0.3.2/dev/yes24) to /tmp/pip-req-build-jzazrjve
[2024-11-21, 05:25:37 UTC] {process_utils.py:194} INFO -   ERROR: Error [Errno 13] Permission denied: 'git' while executing command git version
[2024-11-21, 05:25:37 UTC] {process_utils.py:194} INFO - ERROR: No permission to execute 'git' - install it locally, globally (ask admin), or check your PATH. See possible solutions at https://pip.pypa.io/en/latest/reference/pip_freeze/#fixing-permission-denied.
[2024-11-21, 05:25:37 UTC] {taskinstance.py:3311} ERROR - Task failed with exception

yaml 안에 깃 설치 코드 추가

airflow-worker:
    <<: *airflow-common
    user: root
    command: >
      bash -c "apt update && apt install git && Y && celery worker" #이 부분 추가
      #command: celery worker
    healthcheck:
      # yamllint disable rule:line-length
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    environment:
      <<: *airflow-common-env
      # Required to handle warm shutdown of the celery workers properly
      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
      DUMB_INIT_SETSID: "0"
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

해결완료!