jet / FsKafka

Minimal F# wrappers for Confluent.Kafka+librdkafka.redist 1.x
https://github.com/jet/dotnet-templates
Apache License 2.0
88 stars 17 forks source link

the interval for checking threshold on max inflight counter differs between FsKafka0 and FsKafka #63

Closed wantastic84 closed 4 years ago

wantastic84 commented 4 years ago

Just a small thing I found while doing some research: FsKafka0 has 5ms interval for checking the threshold for max inflight bytes whereas FsKafka has 1ms

FsKafka0: https://github.com/jet/FsKafka/blob/4de31e3662b60b2b0542f121702affa9846bf4ff/src/FsKafka0/FsKafka.fs#L365 FsKafka: https://github.com/jet/FsKafka/blob/4de31e3662b60b2b0542f121702affa9846bf4ff/src/FsKafka/FsKafka.fs#L440

I wonder if this was intended.. Otherwise, we can sync them to be the same (5ms?)

bartelink commented 4 years ago

Consistency is def good; I'm thinking 1ms is best?

My reasoning was that in a scenario where we end up blocking because there's a lot of work on, and that work is large messages that can be processed quickly, we need to resume reading quickly, or we'll oscillate between empty and full frequently if we can get a bufferful done in less than 5ms.

The only reason not to do that is that you're making the reader thread processing hotter and/or triggering more context switches and/or processor cache invalidations - Its definitely likely that there are workloads imaginable where doing shorter (1ms) sleeps can reduce throughput significantly.

(any fix PR should be based on https://github.com/jet/FsKafka/pull/61, which will merge before the next release)