search

airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
630 stars 474 forks source link

Feature - add dask executor support #815

Closed cherusk closed 2 months ago

cherusk commented 6 months ago

Checks

Motivation

I would like to use this chart as part of an step in the efforts of operationalizing godon stack.

I find dask executor is a great option, in my mind even superior to the other executors. In other words I do not want to use the othe executors for various reasons.

Implementation

As several executor "backends" are already supported by this chart, taking up dask executor should not be too great of a lifting.

Are you willing & able to help?

thesuperzapper commented 5 months ago

@cherusk Are you interested in contributing to this feature, or are you only able to test?

Because I am open to any contributions you might have in this space.

cherusk commented 5 months ago

@thesuperzapper I've perused the code an I think I can mostly implement this myself.

Viably I will need supervision/advice beyond a PR review how to approach it best.

I'll file a PR soonish ...

thesuperzapper commented 2 months ago

@cherusk it looks like the upstream project has decided to stop supporting DaskExecutor:

:(

cherusk commented 2 months ago

@thesuperzapper that's only part of the truth. They stopped maintaining/updating it themselves, but it will still work with future versions.

Please see https://github.com/apache/airflow/pull/35965#issuecomment-1882890410

I don't see this to be reason enough to close the PR right away.