Is it possible to mirror a ckan installation via the harvester?

[xellos866]

Hello

This may be a feature request in some way.

I've done some testing while harvesting a ckan system via the "ckan_harvester" plugin. It occurs to us that data won't get deleted or made private when this happens on the target system:

Action on target system	Harvester action as is	Harvester action as assumed
dataset createt	harvest (create)	harvest (create)
dataset updated	updated	updated
dataset removed	no action	removed
dataset created as private	no harvest	no harvest
dataset updated from private to public	harvest	harvest
dataset updated from public to private	no action	removed

First the question if this is the expected behaviour? I can't find a clear definition on your documentation. I would assume now that the goal is to keep the data even if it get's deleted or made private on the target system.

In our case we need a setting to mirror the target system (as assumed in the table above), like:

{
    "mirror_target": true
}

• mirror_target
  • true: removes data if it is not on the target system or made public
  • false: (default): old behaviour, keep all datasets

System:

• Ckan 2.8.3
• ckanext-harvest master@94f0438c0ec6bfd7c85e64a3c15338a4d7800f2c
• ckanext-scheming 1.2.0

[xellos866]

Hello,

little Update: we implemented this behaviour in a custom ckan harvester. The mirror option depends on the "force_all: true" configuration. If you are interested i could create an PR or expose the code some other way (patching of main functions were necessary).

Currently our system uses:

• Ckan 2.8.6
• ckanext-harvest v1.3.1

(our next milestone will be an Upgrade to Ckan 2.9.x)

Greetings

[frafra]

That would be great, thank you :)