Harvest: transition from metadata_review to metadata_extraction or process is a very large request

[atruskie]

I intentionally left the two transitions that re-enqueue all the files for processing as a request that might take a long time.

Essentially both transitions iterate through all harvest items and enqueue a harvest job.

1000 items means the request takes 7.7 second to process. 10,000 items would be 70+ seconds.

7 seconds is quite slow, 70 seconds causes timeouts. We need to improve the amount of time it takes to process these two requests.

When doing this we need to consider what kind of race conditions would be introduced when enqueuing these jobs asynchronously.

[atruskie]

Just did a 10,000 item test on staging. The request took 66 seconds.

We're definitely running close to timeout scenario. Each job only takes 6ms to enqueue, but that adds up!