issues
search
anyscale
/
airflow-provider-ray
Ray provider for Apache Airflow
Apache License 2.0
47
stars
9
forks
source link
Topic/fault tolerance failure poc v2 - DO NOT MERGE
#31
Closed
pgzmnk
closed
3 years ago
pgzmnk
commented
3 years ago
Introduces execute, success, failure, and retry callbacks, which are Native to Airflow Operators
POC on_success_callback retrieves object value from Ray and writes to GCS
POC on_retry_callback executes a function in Ray cluster to fetch GCS object
To-do:
Define callback conditions and execution logic. E.g.
checkpoint=True
Clean up into helper functions
Ensure on_retry_callback passes object to task as argument
Migrate unit tests
To-do:
checkpoint=True