geosolutions-it / oaw-georectify-process-airflow

Airflow based project to manage the geoprectify process
1 stars 0 forks source link

Increase robusteness of DAG's start process #23

Closed chpicone closed 3 years ago

chpicone commented 3 years ago

Currently we create a new DAG for each GeoTif added in the SFTP upload folder. Some times if the GeoTif is big the DAG is created whene the image is still in upload. If the process starts before to be completely uploaded the DAG fails. We need to add an additional (with max retries) step to check if the image is uploaded (eg making a gdalinfo request on it). If the check fails we sleep for some minutes, then retry again.

I suggest these default parameters:

After that if we continue having error we can mark the DAG as failed.