Add Discover short circuit to discover_from_s3 airflow task

NASA-IMPACT / veda-data-airflow

Airflow implementation of ingest pipeline for VEDA STAC data

Other

7 stars 4 forks source link

Add Discover short circuit to discover_from_s3 airflow task #190

Closed amarouane-ABDELHAK closed 2 months ago

amarouane-ABDELHAK commented 2 months ago

Summary: Summary of changes In case discover_from_s3 didn't discover any file form S3 (list is empty) the DAG will gracefully skip all the remaining tasks instead of failing.

amarouane-ABDELHAK commented 2 months ago

Case of no file discovered Case of files discovered