Kafka transactions log constantly growing up

[wooch82]

Issue submitter TODO list

• [X] I've searched for an already existing issues here

Describe the bug (actual behavior)

We are running out of space at the kafka-controller pod in only 3 days and a simple app. During the investigation I noticed that transactions-0 and ingest-transactions-0 group logs are too large.

sentry-kafka-controller-1:/bitnami/kafka/data$ du -sh ./*
...
4.0K    ./bootstrap.checkpoint
0       ./cleaner-offset-checkpoint
102M    ./events-0
12K     ./events-subscription-results-0
72K     ./group-attributes-0
65M     ./ingest-replay-events-0
283M    ./ingest-replay-recordings-0
12K     ./ingest-sessions-0
3.5G    ./ingest-transactions-0
4.0K    ./log-start-offset-checkpoint
4.0K    ./meta.properties
12K     ./profiles-0
12K     ./profiles-call-tree-0
4.0K    ./recovery-point-offset-checkpoint
4.0K    ./replication-offset-checkpoint
12K     ./scheduled-subscriptions-events-0
12K     ./scheduled-subscriptions-generic-metrics-counters-0
12K     ./scheduled-subscriptions-transactions-0
12K     ./shared-resources-usage-0
12K     ./snuba-dead-letter-generic-events-0
12K     ./snuba-dead-letter-replays-0
12K     ./snuba-generic-events-commit-log-0
1.8M    ./snuba-metrics-commit-log-0
12K     ./snuba-queries-0
4.1G    ./transactions-0

There are no problems with the data in the sentry interface and clickhouse table has data as well.

Expected behavior

Data should be cleaned up with the cleaner or retention policy.

values.yaml

kafka: controller: replicaCount: 3 extraEnvVars:

• name: KAFKA_LOG_RETENTION_HOURS value: "24"
• name: KAFKA_LOG_RETENTION_BYTES value: "53687091200"
• name: KAFKA_LOG_SEGMENT_BYTES value: "1073741824"
• name: KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS value: "300000"
• name: KAFKA_LOG_SEGMENT_DELETE_DELAY_MS value: "60000"

Helm chart version

23.4.1

Steps to reproduce

NA

Screenshots

No response

Logs

No response

Additional context

No response

[maitredede]

Hello,

If I remember well, Kafka default retention for topic messages is 7 days (168 hours). So Kafka will not clean messages before retention is reached. Also, I think it does not know if message has been consumed or not, and could be safely deleted. I don't know if compression is activated on these topics, and I don't know what is the cause of so much data. (I use kafka at work, I am new to Sentry). If you are running out of space, I think either you can increase volume sizes (all defaults are around 8-10Gi) or try to reduce ingested data.