search

sentry-kubernetes / charts

Easily deploy Sentry on your Kubernetes Cluster
MIT License
1.02k stars 491 forks source link

Kafka runs out of memory #1345

Open karol-szymanowski opened 3 weeks ago

karol-szymanowski commented 3 weeks ago

Hi,

I am running to a problem where every week kafka is running out of storage space. I run 3x kafka instances each with 100gb storage and it takes just a week to fill the storage. I'd assume kafka is used as a queue for workers and the events are cleared after processing but apparently it's not the case. I usually just clear the storage and reinitialize the kafka cluster which seems to work and I didn't observe any data lose except for events that didn't get into the queue before kafka failure.

Is there any job that I can tweak to clear kafka more often?

kamenskiyyyy commented 2 weeks ago

Hi, I fixed this problem by adding parameters to the config:

kafka:
  extraConfig: |
    log.retention.hours=4

You can experiment with the number of hours to get an optimal result.