Clear worker pools

[martinsumner]

If worker pool queues backup, it may get to the stage where the work is itself wasteful (i.e as the client has already timed out waiting for a response).

The ideal answer is to include the timeout when adding to the queue, so that any work on the queue which is beyond its timeout is discarded not worked. Or allow for a timeout to be set per pool.

In the short term, allowing an operator to clear the queue would be helpful.

@stephenwood24-nhs

[martinsumner]

Thinking about this some more, there is an issue with discarding work. The work may be involve a snapshot encapsulated within a FoldFun. Without calling the fold, the snapshot isn't closed. so it is not as simple as simply clearing the queue.

There was a similar situation with aae_runner queue in kv_index_tictactree, where the fold itself needed to recognise the timeout and close down if it was triggered after the timeout.