search

airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
655 stars 476 forks source link

ExtraPipPackages are supported for 'airflow' values node, not 'scheduler' #888

Open chrisdonlan opened 1 month ago

chrisdonlan commented 1 month ago

Checks

Description

The documentation generated 5 months ago indicates you can modify airflow in the scheduler with extraPipPackages rather than "all" airflow pods by including the scheduler section of values.yaml:

# values.yaml
scheduler:
  extraPipPackages:
    - airflow-provider-foo-bar
    ...

However, that does not work. My mistake here was thinking that meant that extraPipPackages had been deprecated, and I needed to consider other methods such as supporting my own airflow image or adding an init scheme.

After a significant amount of time, I decided to try some of the other nodes to see if the schema would allow extraPipPackages there (in case it was a bug or mistake). The result was good:

# passes schema checks
airflow:
  extraPipPackages:
    - airflow_declarative

airflow: seems like the values node least likely to be supported if a child item is unsupported, since it is the most comprehensive option. But from a testing perspective, it makes sense: it is the first thing you'd try when QA'ing, and the last thing you'd try when using for the first time.

I am not sure whether only scheduler has this option disabled, or everything other than airflow has the option disabled. I can say it took a non trivial amount of time for me to decide the other options weren't worth it and start trying things on the off chance the docs weren't correct.

Are you willing & able to help?

thesuperzapper commented 2 weeks ago

@chrisdonlan I'm confused, what are you having problems with?

I just tested by setting the following values on version 8.9.0 of the chart, and it worked fine:

scheduler:
  extraPipPackages:
    - "tailscale==0.6.1"