search

nansencenter / django-geo-spaas-harvesting

Harvest data into a GeoSPaaS catalog
GNU General Public License v3.0
1 stars 1 forks source link

Add ingestion recovery #126

Closed aperrin66 closed 1 year ago

aperrin66 commented 2 years ago

Resolves #125

Recovery during ingestion: when we don't manage to get the information about a dataset, the info about it is dumped in a file so that it can be retried later. A recovery module to do that has been added and is called at the end of the harvesting process.

If an error happens during the crawling phase, we immediately retry. This can be enhanced later on.

aperrin66 commented 1 year ago

Discussed with @akorosov , merging.