DCAT Harvesting

[trickvi]

Harvest datasets from DCAT sources into CKAN. We'll be following http://spec.datacatalogs.org/ and using https://github.com/ckan/ckanext-dcat

We can assume a /datasets/dcat endpoint to harvest from if we want.

Checklist

• [x] DCAT fields (not just standard CKAN dataset fields) should be harvested. We may also need to harvest DCAT-AP fields not just DCAT ones. Exactly what dataset fields do we need to harvest?
• [x] Harvesting should happen automatically once per week
• [x] How frequently automatic harvesting happens should be configurable. On a per-harvest source basis or just site-wide?
• [x] Datasets that have been deleted from the source site should be deleted from CKAN the next time that site is harvested
• [x] Harvested datasets should be immutable - can only be changed by re-harvesting (if the source dataset has changed)
• [x] Users need to be able to add new URLs to harvest from (the CKAN harvester already supports this)

[trickvi]

@seanh

The clarified DCAT AP expression which @matthiaspalmer has been writing should list what dataset fields we should harvest.

The configuration of the automatic harvesting should just be a site-wide one. That's simplest.

[trickvi]

@seanh You said previously:

"The CKAN harvester already provides an admin interface where users can give the URLs to be harvested from. The OrgResp will need to have an admin account on the CKAN site."

This is not the same as a sysadmin account right? If that's the case we should look into opening up the permissions somehow.

[seanh]

I think Adria is going to be working on this..