Save CKAN ID to HarvestRecord

[btylerburton]

User Story

In order to keep track of packages once they are created in CKAN, datagovteam wants to store the ckan_id which is returned after package_create.

This will then be utilized on package_update and package_delete.

Acceptance Criteria

• [ ] GIVEN I have created a package in CKAN WHEN CKAN returns the id of the new package THEN I want to store that package id on the record for retrieval later when future update/delete operations on that same dataset are needed.

• [ ] GIVEN I want to update a package in CKAN WHEN I pass the updated ckanified_metadata THEN I need to include the ckan_id

• [ ] GIVEN I want to delete a package in CKAN THEN I need to pass id=ckan_id

Background

To delete or update the dataset, the ckan package id is needed.

Security Considerations (required)

None

Sketch

• [ ] get ckan dataset id returned after running package_create
• [ ] store that on harvest_record
• [ ] query original record prior to invoking the update/delete operation.
  • [ ] note that when new HarvestRecord objects are created from the internal records that this id is not getting passed.

[btylerburton]

We could use either below for key relationships:

• harvest_record.id
• harvest_source.id:harvest_record.identifier

I was leaning towards the former as it's 1:1, but if we're updating a dataset multiple times you'd get multiple entries in the lookup table which would then necessitate a date to sort them by.

Does it make better sense to use harvest_source.id:harvest_record.identifier?

1. No lookups. We have both those values available at time we'd need to use them.
2. Unless the identifier changed we have an entry pre-existing that we could just defer to after the create
3. BUT ... If a data provider shuffles identifers around, we could end up with things in a disconnected state.

[rshewitt]

the harvest record table has a ckan_id field. we could include the ckan id in our sync success update. once we get the id from ckan after creation I would expect it wouldn't change even with an update? so we could add a ckan_id attribute to the record instance, assign the ckan id to the record instance on package create, then check if that value is not none in the update self in db method and add it to the update dict?

[btylerburton]

the harvest record table has a ckan_id field.

That it does! Ok. Now we'd just need code changes to:

• grab that ckan_id after package_create and sync it to the record
• make that available to update/delete, either by attaching it to the new record that gets created in internal_records_to_id_hash or by stashing that somewhere in a temp lookup table on the harvest_source.

[Jin-Sun-tts]

Retrieved the ckan_id from the CKAN package_create API call.

Tested it in development, deleted the all dataset in test org and clean the records in harvestDB. Re-harvest and the ckan_id is updated in the harvest_record table.