search

dpkp / kafka-python

Python client for Apache Kafka
http://kafka-python.readthedocs.io/
Apache License 2.0
5.64k stars 1.41k forks source link

After consuming message new topic __consumet_offsets is created in the Kafka Broker #581

Closed iammehrabalam closed 8 years ago

iammehrabalam commented 8 years ago

New topic name __consumer_offsets is created when consumer start consuming messages. Topic is created with 50 partition and each partition takes lot of space around GigaBytes to maintain two files .index and .log.

Please tell me the reason why we need to create new topic in Kafka broker and why it is taking lot of space?

my configuration

{'enable_auto_commit': True, 'auto_commit_interval_ms': 100, 'auto_offset_reset': 'largest', 'bootstrap_servers': KAFKA_BROKER_HOSTS, 'group_id': 'something' }

dpkp commented 8 years ago

This is a kafka server feature (not specific to kafka-python). See http://kafka.apache.org/documentation.html#impl_offsettracking . For more information about kafka broker operations, I suggest joining the kafka users mailing list. More info at https://kafka.apache.org/contact.html