search

mgalas / OpenDataHub

Master repository for all the open source projects driven by UDL
GNU General Public License v3.0
3 stars 4 forks source link

CKAN harvesting for data.gov.uk #3

Open mgalas opened 7 years ago

mgalas commented 7 years ago

Enable CKAN data harvesting for data.gov.uk

metaljovi commented 7 years ago

already enabled we can harvest from another CKAN instances (such as data.gov.uk) and also from CSW server and WAF

But probably need some workaround so the harvester can continously work from background at the moment I have to start it manually

mgalas commented 7 years ago

@metaljovi absolutely agree on the scheduled process approach. Is there anything of-the-shelf that we can use or do we need to script something ourselves (e.g. CRON + BASH script)?

metaljovi commented 7 years ago

we can use CRON and supervisor from what I know, there is already tutorial about this actually I already tried that and it worked

The problem raises when I tried to add a new sources when the harvesting runs in the background, the harvester still only harvest from the existing sources and not start harvest from the new sources.

mgalas commented 7 years ago

hmm in theory this is fine for the "currently active" run to harvest what was intended at the time when the process started. As long as it harvests the new sources the next time the process starts it should be fine

mgalas commented 7 years ago

BTW if you'll go for the CRON approach then please put scripts into /src/main/resources of the project for consistency