Both settings help to influence the write load that hits the sink data store but at the same time it is not flexible enough in order to achieve a somewhat predictable rate limiting behaviour. A new configuration option could offer something similar to the following:
Introduce a timeout parameter specified either in long ms or maybe in Java 8 style Durations. This would introduce an "artificial pause" for the worker thread during the put() call - either directly or by signalling it to the sink connectors context.
Introduce a related parameter (int n >= 1) which specifies after how many put() calls successfully processes batches within one and the same put() call the timeout should have an effect e.g. every N-th batch for a topic/collection within the put() call "pause" for 100 ms
hpgrahsl
commented
5 years ago
Actual throughput rate of the sink connector can already be controlled at least to some degree:
Both settings help to influence the write load that hits the sink data store but at the same time it is not flexible enough in order to achieve a somewhat predictable rate limiting behaviour. A new configuration option could offer something similar to the following:
Introduce a timeout parameter specified either in long ms
or maybe in Java 8 style Durations. This would introduce an "artificial pause" for the worker thread during the put() call - either directlyor by signalling it to the sink connectors context.Introduce a related parameter (int n >= 1) which specifies after how many
put() callssuccessfully processes batches within one and the same put() call the timeout should have an effect e.g. every N-th batch for a topic/collection within the put() call "pause" for 100 ms