Determine viability of replacing datapusher with xloader

[hleckenb]

---

User Story

As a BC Data Catalogue Administrator,
I want to ensure that data pushes to the Datastore succeed
**so that data loads can be trusted and not end up with partial/half loaded data or requiring manual or further administrative tasks to successfully push data to the Datastore.

---

Test Case

ENV

PROD, CAT, CAD **please verify environment

TESTCASE

*

EXPECTED

*

ACTUAL

*Datapusher creates 502 error As Dave describes below, when you do an Update Dataset the update fires a trigger that

• Deletes each resource for the record from the datastore
• Repushes the resource from the filestore

From the logs it appears that it is this trigger that is running into problems. Specifically the trigger thinks that a resource is in the datastore, it tries to delete it and doesn’t handle the fact that it doesn’t exist very well. Why doesn’t it exist? In my testing I made sure that all the resources were in the datastore (by previewing them) before I did the Update Dataset. SO what happened to the resources that can be found? Could there be a competing process that has deleted them before the trigger tries to delete them? (i.e., a “race condition”). We also know that turning off the data pusher avoids the problem. Turning off the data pusher means that the trigger won’t be able to do anything and hence won’t get into trouble.

“Guessed” data type turning out to be wrong halfway through the file. There might be an issue with multiple heading lines. To fix the record that came up yesterday (with its 20 or so resources) will take the better part of a day, and Leo’s involvement, and the involvement of the data provider.

[hleckenb]

Investigate - Implemented xloader may remove/reduce error return. https://github.com/ckan/ckan/issues/4415

[hleckenb]

@dkelsey please complete the test case for this.

[dkelsey]

• @hleckenb I'm going to assume you want a test case for what we're calling "the 502" error?
• That is a separate issue from this.
• This ticket should be a user story that describes what a user expects when they upload a file(?)... they expect it to be loaded into the DataStore.
• We do need a way to express "we need to do work exploring using xloader"
• Are we proposing we use xLoader because it delivers more value to users? or are we "hoping" that is will resolve the "502" issue?

I have more to say but i'll stop there.