Temporarily override delay setting on a disconnected pipeline

[JustAnotherArchivist]

Yesterday, one pipeline suddenly disconnected due to an emergency network maintenance. It would be useful if the operator (assuming access in some other way is still possible) could adjust the delay between requests in such a case until network is restored to avoid the queue draining with timeout errors. This could e.g. be a file which is monitored by the settings updater like it's done in grab-site. (Normal delay changes are obviously not possible because the job can't read the new settings from Redis.)

However, this would lead to an inconsistency in what the job is actually doing vs. what the control node thinks its settings are. I'm not sure how to best resolve that. The pipeline could write back the new delay value to Redis upon reconnecting, and the IRC bot could then announce it on the channel. Or the pipeline could simply give priority to Redis, i.e. prefer the value from Redis if the connection to the control node is up.

[kiska3]

The pipelines which disconnected for context: kiskaAus - pipeline:eaa5833a8b0fa8e939365218721bf19b kiskaAus-2 - pipeline:198f51cd600240b984db7a3285cdb9f5 kiskaAus-3 - pipeline:9e4630463fcbe8e3f08e7334e128217d kiskaAus-4 - pipeline:4deeb06194cd01a1ce54e4f39ffc0409 kiskaAusAO - pipeline:6b87cd42d3e014d38b38d06fb3d1b926 kiskaAus-con1 - pipeline:1a563314fd8a815432a5ac71d3b2ffd8

The disconnect was due to an unscheduled maintenance on the nbn(National Broadband Network) which was outside of ISP control, in which the HFC node suddenly lost power and was unable to reconnect for some time.