Hydroponics does not update its enabled queues list while already running

[DiegoPino]

What?

Once Hydroponics quicks in Queues marked to be processed by it are read once and used. Any attempt to disable a queue (e.g you know you want to stop processing for a while the background one) will have no effect. That is not ideal.

At least on each Hydroponics round robin (means when it swaps context from one queue to the other) we should re-check what queues are enabled to respect configuration changes

@alliomeria I just noticed this today. While it was running I disabled the Background (OCR) one but OCR kept being executed.

It would be also idea to have a "stop/you did enough" state setting that is not killing Hydroponics but allows it to shutdown gracefully after it finishes one item. Would be ideal that also a "time to live" update could trigger a change. If set to live "0" means until finish setting 720 should change the scope and allow Hydroponics to shutdown sooner.

Lastly:

• we should keep track of stats (time per queue item) after each Hydroponics wake up/shutdown cycle and allow that data to be reported to the user in the hydroponics page as stats on each queue listing
• we should have a config (in case someone gets creative, have seen it) that sets how many Hydroponics services can even run on a server. We don't allow/can/will run more than one, but if you call the drush script manually via an external Service Cron you might have multiple ones. It WON'T HURT if it runs (it was designed to be atomic). Queue items are independent but not every server can ingest/OCR/etc 3 or 4 at the same time without becoming a pile of ashes and black smoke. Just saying.

[dmer]

This is excellent! Super useful tools for repo admins trying to manage large scale ingest/migration. We have been negotiating with Hydroponics a lot in the CAR site - upping the system resources has helped, but I have been wishing for more visibility into what it's doing, when as well as a teeny bit more control over gracefully pausing/stopping things to sort out an issue or allow some other big process to happen. Stats would also help in providing folks with better prediction and better diagnostics. Pat's away this week, but I will look at this closely w/ him to see if we can think of any other suggestions for this area. Thanks!