search

ngds / ckanext-ngds-bku03232018

CKAN extension for NGDS-specific customizations
Other
5 stars 13 forks source link

Harvester screen out duplicate records based on gmd:fileIdentifier #636

Open smrgeoinfo opened 8 years ago

smrgeoinfo commented 8 years ago

If a harvested record has a fileIdentifier that is already in the harvested table (from any harvest source), then check

  1. which record is newer
  2. do the links work use newer record if its distribution links work, or none of the links work use older record if newer one has broken links and older record has links that work

this is part of https://github.com/ngds/ckanext-metadata/issues/23

GaryHudmanAZGS commented 8 years ago

Have implemented the harvester checks for all guids in database, using newest record. Tested and in production. Currently the resource link check quality control is separate process that is run periodically.