search

confluentinc / parallel-consumer

Parallel Apache Kafka client wrapper with per message ACK, client side queueing, a simpler consumer/producer API with key concurrency and extendable non-blocking IO processing.
https://confluent.io/confluent-accelerators/#parallel-consumer
Apache License 2.0
94 stars 134 forks source link

Add distributed rate limiting support #24

Closed astubbs closed 1 year ago

astubbs commented 4 years ago

Use a central cache to share rate limit info amongst processing cluster, to implement coordination of rate limit exhaustion to prevent accidental overrun while effectively exhausting the available credits.

https://dzone.com/articles/introduction-jcache-jsr-107 https://github.com/vladimir-bukhtoyarov/bucket4j vs https://github.com/mokies/ratelimitj

resilience4j doesn't support distriubted rate limiting: https://github.com/resilience4j/resilience4j/issues/350#issuecomment-720961836

astubbs commented 3 years ago

Had a quick look at it - something like this with Reactor and BucketJ should work.

johnbyrnejb commented 1 year ago

Closing Issue