split harvester batch process into more manageable chunks

jvitello commented 7 years ago

Description

The harvester GUI uses a batch process, but it completes it in one step from 0% to 100%. We have been experiencing errors when running the harvester via the GUI that seem to be related to the batch being one big process (time outs and server unavailable errors).

We think it will be a better user experience for incremental progress to be displayed. We think some of the other tickets in the backlog may also be resolved with this change.

The harvester also fails after long harvester sessions. The hope is that this will resolve those issues as well.

Steps to Reproduce

add https://data.chhs.ca.gov/data.json as a harvest source (it's a big source at 262 datasets)
go to the source page, preview and harvest now
also test migrations via the dkan harvest dashboard

Dev Notes

We need to figure out how much the batch should be divided. Maybe every 20 datasets per chunk.

Acceptance Criteria

batch process is broken into smaller chunks
after each chunk is processed, the batch progress bar is updated for the user

jvitello commented 7 years ago

@dangur Were you able to start this issue?

dangur commented 7 years ago

@jvitello not significantly

jvitello commented 7 years ago

@dangur If no one has really started, we will probably take this one back. It's becoming a higher priority.

dangur commented 7 years ago

That is fine. Please reassign to someone who can work on it soon.

GetDKAN / dkan