[mget] Poll for tasks less frequently when the task load doesn't need it

[mikecote]

When enabling mget as the task claim strategy, the task poll interval currently changes from every 3s to every 500ms, creating 6x the requests to Elasticsearch to claim tasks. Lowering the poll interval was added to increase the per-node task throughput. However, when observing the request load to Elasticsearch for small serverless projects that don't run many tasks, the Task Manager performs a lot of requests for little of a return.

It would be nice to poll less frequently whenever the task load doesn't need to, say back to 3s in such situations. We should take the following into consideration when coming up with an approach:

• Backpressure mechanism: we'll need to continue applying backpressure whenever 429 errors are observed
• Task distribution: we'll need to ensure the Kibana nodes are still running tasks evenly so one doesn't poll frequently and run all the tasks while the other doesn't poll frequently and doesn't find many tasks to run

Definition of Done

• [ ] Poll interval is 3s whenever the task load doesn't need a 500ms poll interval
• [ ] Backpressure mechanism still works when encountering 429 errors
• [ ] Tests

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

Seems like it would be nice to have all the claiming-rate bits in one place - it gets fed various inputs (the 429's etc, and now number of tasks claimed over last cycle(s)) - and generates a new rate. So we don't have the logic spread out all over the place ...

[pmuellr]

Another thing we can do is check for upcoming tasks to run - nothing? Then sleep a bit longer ...

Maybe the claimer could search a bit in the future as well, and determine if there is anything coming up soon ...