Re-enable resubmit_jobs in ckanext-harvest

GSA / data.gov

Main repository for the data.gov service

https://data.gov

Other

582 stars 91 forks source link

Re-enable resubmit_jobs in ckanext-harvest #671

Open adborden opened 5 years ago

adborden commented 5 years ago

ckanext-harvest upstream has resubmit_jobs as part of harvest run. This requeues harvest tasks that have timed out. The upstream timeout is 3 minutes, which might be too short for us. I would be happy with a 60 minute timeout. We should update the timeout and re-enable resubmit_jobs.

adborden commented 5 years ago

This is also related to fixing the duplicates issue in ckanext-datajson. Having the ability to requeue tasks would simplify how we process parent and child datasets.