search

geosolutions-it / evo-odas

Code Repository for the EVO-ODAS
https://waffle.io/geosolutions-it/evo-odas
MIT License
31 stars 15 forks source link

DAGs exit immediately when there are no products to process #245

Open randomorder opened 6 years ago

randomorder commented 6 years ago

Would be nice to have DAGs runs exit immediately after the search / download tasks is there is no products to process.

Currently this does not happen and all tasks are executed anyway. Operators should handle this gracefully as per #243 but I think would be good to have the DAGs terminate their execution early when there is nothing to process.

One way to achieve this would be to branch after the search / download and have a Dummy "finish" task by changing existing DAGs:

One possibility is to branch after the search / download and have a Dummy "finish" task:

From graph1

Into: graph2

Airflow may also provide other mechanisms to do this

@torse What do you think

torse commented 6 years ago

Maybe we could branch the DAG before Download? Otherwise LGTM.

I like the approach to define a Finishing task. It could later on also send notification and/or update the elasticsearch database..

maabdelghaffar commented 6 years ago

@torse @randomorder how about using ShortCircuitOperator to allow the DAG to continue running only if our condition met ?

randomorder commented 6 years ago

I don't have a strong opinion on this either would work for me