DCAT-US Load Test in Cloud.gov

[btylerburton]

User Story

In order to properly assess progress and define bugs in the new harvester, datagovteam wants to load test some DCAT-US sources in Cloud.gov.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• [ ] GIVEN I have configured an organization and a harvest source in the datagov-harvester-admin AND I have created the same organization in the corresponding CKAN environment WHEN I trigger a harvest in the admin app THEN I expect on refresh of the application that the LoadManager would schedule a new harvest run

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• [ ] Confirm datagov-harvester-admin and datagov-harvester-runner are deployed to the latest version
• [ ] Confirm a Test Organization and Harvest Source are configured in datagov-harvester-admin
• [ ] Confirm the same Test Organization is configured at the corresponding CKAN instance
• [ ] Confirm the datasets enumerated in the configured harvest source URL are not already posted to the corresponding CKAN instance
• [ ] Trigger a new harvest in datagov-harvester-admin
• [ ] Restart the datagov-harvester-admin application to kick off the load manager

[btylerburton]

Load test daily EOD:

• I need ckan id to delete a dataset. We aren't storing that. Should we store that on create/update just to make things easier?
• If a delete fails, we mark it as an error, but what we need to recreate current state of CKAN source is packages with action: create/update, state: success AND action: delete, state: error