Throttling control during execution

[soyacz]

There's an idea to add api endpoint to running cql-stress command that can adjust throttling during the run.

This feature would enable us to actively adjust load inflicted to cluster based on custom logic. For example, increase/decrease ops to achieve specific Scylla reactor load (50-75% for typical longevity test, 100% for SLA tests), so our tests are less affected by machine types used and speed up test design process by eliminating need for manual fine-tune loader commands. Feature will enable us also to define custom load patterns (like ramp up, spikes, find break-point when latency exceeds given value).

There's also idea to make feedback loop to be fully controlled by cql-stress (e.g. following certain Scylla metric), so it's possible to run cql-stress without any external processes that control throttling - for ease of use in typical cases. (My proposal is to do in follow up tasks with specific scenarios. Still we need to experiment with it, especially in multi-loader, many load processes tests).

[soyacz]

cc @fruch @roydahan

[roydahan]

When reaching this one, it'll require a design meeting.