search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.52k stars 14.14k forks source link

Support hybrid executors in backfill jobs #40472

Closed o-nikolas closed 2 months ago

o-nikolas commented 3 months ago

The backfill job does its own scheduling and so needs changes to support hybrid executors outside of the Airflow Scheduler.

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk commented 3 months ago

Looks good. That's pretty non-obvious complexity with hybrid executors :)

kaxil commented 2 months ago

@o-nikolas Can this be merged?