search

infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
https://www.fluvio.io/
Apache License 2.0
3.88k stars 491 forks source link

Throughput Control #2512

Closed galibey closed 2 years ago

galibey commented 2 years ago

We need to find a way to control the throughput that we produce during integration testing. Currently, it is limited only by available system resources and is unstable due to that fact.

We need to figure out on which level we want to do this. Possible options: fluvio-test where we generate the test data or fluvio client where we can add a back pressure mechanism to control the throughput that goes to the Producer.

tjtelan commented 2 years ago

I vote for adding this in fluvio, and adding a fluvio-test test to externally verify the throughput based on client configuration.

Users would want this, and it would be nice to separate our need for client back pressure support from fluvio-test.

sehz commented 2 years ago

We need to fix this ASAP with CI becoming more flaky