search

GSA / data.gov

Main repository for the data.gov service
https://data.gov
Other
634 stars 100 forks source link

Harden the CSW harvester (failed requests derail harvests) #672

Closed adborden closed 4 weeks ago

adborden commented 5 years ago

The CSW harvester keeps failing for NOAA. The gather stage has to fetch thousands of records and any 500 on their side will fail the job. We need to review the CSW, add some retry logic, and do some general hardening to so the harvest is more robust.

mogul commented 4 years ago

NOAA will be rescued from this state by the work in #1419, but we're leaving this issue open as there are likely other agencies affected and the problem still exists.

jbrown-xentity commented 4 weeks ago

No longer investing in the CKAN harvester, closing.