search

kafka-dev / kafka

A distributed publish/subscribe messaging service
http://sna-projects.com/kafka
Apache License 2.0
549 stars 300 forks source link

__consumer_offsets topic with very big partitions #75

Open emy-lee opened 3 years ago

emy-lee commented 3 years ago

I am using Kafka 2.0.0 There are some partitions of the __consumer_offsets topic that are 500-700 GB and more than 5000-7000 segments. These segments are older than 2-3 months. There aren't errors in the logs and that topic is COMPACT as default.

What could be the problem? Maybe a config or a consumer problem? or maybe a bug of kafka 2.0.0? What checks could I do?

My settings:

log.cleaner.enable=true
log.cleanup.policy = [delete]
log.retention.bytes = -1
log.segment.bytes = 268435456
log.retention.hours = 72
log.retention.check.interval.ms = 300000
...
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 10080
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 3
offsets.topic.segment.bytes = 104857600