Closed Zarathustra2 closed 2 years ago
Hi @Zarathustra2 ! Good question. IIRC each local worker gets its own process - so even if you have a single topic and multiple partitions get assigned to the same consumer node, each partition will get its own process. This is by design for scaling purposes. Multiple topics is just a special case of that I think.
I think the best thing to do here is to have your own shared state management unless there's some reason why that's not practical? I don't really want to go against the scaling structure we have, and any shared state solution we come up with might make assumptions that don't work for all users.
Hey @dantswain thanks for the response!
Yeah I think it is not really a big issue, was just curious. There are enough workarounds for this problem so not having it is not a big deal IMO.
I just thought that the design was the exact opposite until I digged into the code! :)
I think this can be closed now, unless you want to leave it open
OK, thanks! I'll close it so that I can feel like I accomplished something :Joy:
If I start the following consumer group:
If
FooBar
has the following handler:then each topic will have its own state.
Is there a way that one consumer can consume from multiple topics but share the state? It looks like one consumer module is being started for each topic.
Best regards, Dario