Currently we create a new DAG for each GeoTif added in the SFTP upload folder.
Some times if the GeoTif is big the DAG is created whene the image is still in upload.
If the process starts before to be completely uploaded the DAG fails.
We need to add an additional (with max retries) step to check if the image is uploaded (eg making a gdalinfo request on it).
If the check fails we sleep for some minutes, then retry again.
I suggest these default parameters:
sleep time: 3 min
max reties: 5
After that if we continue having error we can mark the DAG as failed.
Currently we create a new DAG for each GeoTif added in the SFTP upload folder. Some times if the GeoTif is big the DAG is created whene the image is still in upload. If the process starts before to be completely uploaded the DAG fails. We need to add an additional (with max retries) step to check if the image is uploaded (eg making a gdalinfo request on it). If the check fails we sleep for some minutes, then retry again.
I suggest these default parameters:
After that if we continue having error we can mark the DAG as failed.