search

airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
630 stars 474 forks source link

feat: add support for `DaskExecutor` #816

Closed cherusk closed 2 months ago

cherusk commented 5 months ago

What issues does your PR fix?

What does your PR do?

This PR brings about a new Airflow executor backend on Dask/Distributed cluster.

Checklist

For all Pull Requests

cherusk commented 5 months ago

@thesuperzapper this should be it. Please give it a gander for now.

I've not tested this outside of this PR yet.

cherusk commented 5 months ago

@thesuperzapper @gsemet could you please prioritize this up? Thanks!

stale[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had activity in 60 days. It will be closed in 7 days if no further activity occurs.

Thank you for your contributions.

Issues never become stale if any of the following is true:

  1. they are added to a Project
  2. they are added to a Milestone
  3. they have the lifecycle/frozen label
thesuperzapper commented 2 months ago

@cherusk thanks for the PR, but it looks like as of Airflow 2.9.0, the upstream project has decided to stop supporting DaskExecutor:

I understand that will be frustrating!