search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.89k stars 9.16k forks source link

Installing airflow with extra packages #2588

Closed JK87iab closed 4 years ago

JK87iab commented 4 years ago

Which chart: airflow

What is the best practice to install additional airflow and python packages? (https://airflow.apache.org/docs/stable/installation.html)

Is the only approach to rebuild the images?

javsalgar commented 4 years ago

Hi,

We added this to the container so you could add any other python packages

https://github.com/bitnami/bitnami-docker-airflow/blob/master/1/debian-10/rootfs/run.sh#L14

If you mount a requirements.txt pip will be executed. Would that work for your case?

JK87iab commented 4 years ago

Thx that worked

javsalgar commented 4 years ago

Good to know! Do not hesitate to open a new ticket if you find more issues :D

ashishmgofficial commented 3 years ago

@javsalgar The requirements.txt is working great in scheduler but the in worker pods im getting module not found error ? Anything im missing ?

javsalgar commented 3 years ago

Hi,

This is weird, as I see the same code in the worker container.

https://github.com/bitnami/bitnami-docker-airflow-worker/blob/master/2/debian-10/rootfs/app-entrypoint.sh#L17

Are you installing the exact same packages in one and the other?

ashishmgofficial commented 3 years ago

@javsalgar I have a PV created for holding the requirements.txt. so all the worker and scheduler takes the same requirements.txt

javsalgar commented 3 years ago

Hi,

Could you also check if the pip install command is being run in both nodes? If it says module not found error it could be related to the module not being installed in the proper folder (which I don't think it's the case) or maybe something incorrect with the PYTHONPATH variable. At which point does it fail? Which is the exact traceback?

elvin-han commented 3 years ago

Hi, I have same problem.

The requirements.txt is working great in scheduler and web. But not working in worker.

web:
  baseUrl: http://localhost:8080

  extraVolumeMounts:
    - name: requirements
      mountPath: /bitnami/python/

  extraVolumes:
    - name: requirements
      configMap:
        name: requirements

scheduler:
  extraVolumeMounts:
    - name: requirements
      mountPath: /bitnami/python/

  extraVolumes:
    - name: requirements
      configMap:
        name: requirements

worker:
  extraVolumeMounts:
    - name: requirements
      mountPath: /bitnami/python/

  extraVolumes:
    - name: requirements
      configMap:
        name: requirements
elvin-han commented 3 years ago
스크린샷 2021-03-29 오후 7 33 10 스크린샷 2021-03-29 오후 7 34 17
javsalgar commented 3 years ago

Pinging @miguelaeh so he takes this into account when working on the latest chart changes

amal209 commented 2 years ago

Hiii, i created a volume for requirements.txt and upgraded the chart, but i'm still getting ModuleNotFoundError: No module named 'yfinance'

javsalgar commented 2 years ago

Hi,

Could you show the requirements.txt file and the values you used to install the chart?

amal209 commented 2 years ago

here is my requirements and values image

amal209 commented 2 years ago

UPDATE !! I solved the problem by adding extraVolumeMounts and extraVolumes to the airflow web and scheduler pods

amal209 commented 1 year ago

Hello, here's my requirements.txt [image: image.png] And i added this in my values.yaml [image: image.png]

Le mar. 12 avr. 2022 à 08:32, Javier J. Salmerón-García < @.***> a écrit :

Hi,

Could you show the requirements.txt file and the values you used to install the chart?

— Reply to this email directly, view it on GitHub https://github.com/bitnami/charts/issues/2588#issuecomment-1096357563, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANTSYJAOCN2EJAW5WIKGXNDVEUYKNANCNFSM4NA3G26Q . You are receiving this because you commented.Message ID: @.***>

jamesmedice commented 9 months ago

adding extraVolumeMounts and extraVolumes to the airflow web and scheduler pods

in my case I have,

      containers: 
        - name: airflow-web
          image: bitnami/airflow:2.2.3-debian-10-r45
          imagePullPolicy: "IfNotPresent"
          env: 
          envFrom:
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: requirements
              mountPath: /bitnami/python/requirements.txt              
              subPath: requirements.txt
          resources:
            limits: {}
            requests: {}
      volumes:
        - name: requirements
          configMap:
            name: airflow-requirements

how would the extraVolumeMounts and extraVolumes ? should I replace volumes and volumemounts?

vorandrew commented 2 months ago

Why another user is doing pip install ?

airflow-worker 15:55:42.14 INFO ==>
airflow-worker 15:55:42.14 INFO ==> Welcome to the Bitnami airflow-worker container
airflow-worker 15:55:42.15 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-worker 15:55:42.15 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-worker 15:55:42.15 INFO ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
airflow-worker 15:55:42.15 INFO ==>
airflow-worker 15:55:42.16 INFO ==> Enabling non-root system user with nss_wrapper
WARNING: The directory '/opt/bitnami/airflow/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting corsound-airflow@ https://_json_key_base64:****@us-python.pkg.dev/mvp-2023-10-10/corsound/corsound-airflow/corsound_airflow-2.0.0-py3-none-any.whl (from -r /bitnami/python/requirements.txt (line 2))
Downloading https://_json_key_base64:****@us-python.pkg.dev/mvp-2023-10-10/corsound/corsound-airflow/corsound_airflow-2.0.0-py3-none-any.whl (4.5 kB)
Requirement already satisfied: flask in /opt/bitnami/airflow/venv/lib/python3.11/site-packages (from -r /bitnami/python/requirements.txt (line 1)) (2.2.5)
Requirement already satisfied: Werkzeug>=2.2.2 in /opt/bitnami/airflow/venv/lib/python3.11/site-packages (from flask->-r /bitnami/python/requirements.txt (line 1)) (2.2.3)
Requirement already satisfied: Jinja2>=3.0 in /opt/bitnami/airflow/venv/lib/python3.11/site-packages (from flask->-r /bitnami/python/requirements.txt (line 1)) (3.1.4)
Requirement already satisfied: itsdangerous>=2.0 in /opt/bitnami/airflow/venv/lib/python3.11/site-packages (from flask->-r /bitnami/python/requirements.txt (line 1)) (2.2.0)
Requirement already satisfied: click>=8.0 in /opt/bitnami/airflow/venv/lib/python3.11/site-packages (from flask->-r /bitnami/python/requirements.txt (line 1)) (8.1.7)
Collecting peppercorn (from corsound-airflow@ https://_json_key_base64:=@us-python.pkg.dev/mvp-2023-10-10/corsound/corsound-airflow/corsound_airflow-2.0.0-py3-none-any.whl->-r /bitnami/python/requirements.txt (line 2))
Downloading peppercorn-0.6-py3-none-any.whl.metadata (3.4 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/bitnami/airflow/venv/lib/python3.11/site-packages (from Jinja2>=3.0->flask->-r /bitnami/python/requirements.txt (line 1)) (2.1.5)
Downloading peppercorn-0.6-py3-none-any.whl (4.8 kB)
Installing collected packages: peppercorn, corsound-airflow
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/opt/bitnami/airflow/venv/lib/python3.11/site-packages/peppercorn'
[notice] A new release of pip is available: 24.0 -> 24.1.2
[notice] To update, run: pip install --upgrade pip