bitbouncer / kspp

A high performance/ real-time C++ Kafka streams framework (C++17)
Boost Software License 1.0
112 stars 24 forks source link

Multi instance apps should share consumer group #14

Closed krojew closed 6 years ago

krojew commented 6 years ago

At this moment app_info assumes multi instance apps are in different consumer groups, which makes them not scalable.

skarlsson commented 6 years ago

Maybe its bad documentation, but the third argument is not meant for instance since I really mean that the program should take the handled partitions as arguments (or handle all - which does not scale). I've had the use case that a single app handles different "tenants" not partitions - in that case the third argument makes sense. No automatic scaling a'la kafka streams is implemented. Partly because I wanted to avoid the penalty of reassigning state stores when you add a new instance.

krojew commented 6 years ago

Ah, so multi instance should rather be multi tenant. I suggest changing/adding some comments about it. Automatic scaling would be a good idea, tho.