Reader: Changing topic subscription at runtime

[godwhoa]

Describe the solution you'd like Currently, we have to re-create a Reader if we want to subscribe to a new topic, this is perfectly fine when only subscribed to a single topic. But once we have support for consuming from multiple topics (#600), being able to change the subscription on the fly would be nice to have. I think people asking for the topic discovery feature would also want this feature (#408).

Supporting documentation https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe-java.util.Collection- "Consumer Groups and Topic Subscriptions" section at the top (no anchor link for it sadly) https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

CC: @dominicbarnes

[dominicbarnes]

Changing on the fly is not currently possible, but you can subscribe to multiple topics at once when using a consumer group (as of v0.4.12), does that fit your use-case enough?

[godwhoa]

I can elaborate a bit on the use case. Say we subscribe to multiple topics based on a regex pattern and we have topics being created over time. I would want to update my subscription when I find new matching topics on the fly. Being able to update subscription on the fly is the only missing feature to implement this.

[chenzhiwei]

I also have a use case on this, for some reasons, our client needs to listen on a topic pattern, something like com.*, new topics will be created after we started our consumer.

E.g. Before we start our consumer, we have two topics com.github and com.gitlab, another topic com.gitea is created after we started our consumer.

We don't want to restart our consumer every time a new topic is created.

[VRmnv]

Any updates on this?

[godwhoa]

I'd recommend franz-go if you want a more feature complete kafka client.