search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
52 stars 857 forks source link

Question : Kakfa system topic per client #160

Closed eyantiful closed 7 years ago

eyantiful commented 7 years ago

Hi, In the Use case of simple Pub-Sub each consumer is given a unique id (new GUID), Will Kafka create new system topic each time(for offsets)? If we refrain from committing the offset will Kafka clean the clutter? Is there another best practice to how to implement Pub-Sub with Kafka?

Thanks,

mhowlett commented 7 years ago

Offsets for all consumers in all consumer groups are stored by Kafka in a single topic called __consumer_offsets. The broker config parameter offsets.retention.minutes (defaults 1440) controls how long committed offsets are retained in that topic.

Separately, Kafka manages consumer group membership. If all consumers in a consumer group are stopped, the consumer group will be cleaned up automatically (except that any committed offsets related to the group will hang around in the __consumer_offsets topic for the time period noted above).