fluent-bit memory not released

[littlejoyo]

Bug Report

Describe the bug I have a problem about memory, my fluent-bit is enabled jemalloc, I found that when fluent-bit processes logs, the memory usage will gradually increase, and when the traffic stops, I will find that the memory is not released immediately, is this part normal? Is it related to jemalloc?

After fluent-bit processed the log, I found that the memory usage was about 340Mi, and then I stopped the traffic and saw that the memory usage was still about 340Mi the next day, why wouldn't fluent-bit release the occupied memory? Although the data is no longer processed

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 206 root 20 0 6295508 346876 6424 S 0.7 1.1 73:17.02 fluent-bit

Your Environment

• Version used: v2.1.10
• Configuration:

  @INCLUDE ./filters/myconf.conf
  [SERVICE]
  flush                     1
  daemon                    off
  log_level                 info
  plugins_file              ../static/flb/plugins.conf
  parsers_file              ../static/flb/parsers.conf
  http_server               on
  http_listen               0.0.0.0
  http_port                 2021
  health_check              on
  storage.path              /flb/storage/tpla-transform-0
  storage.metrics           on
  storage.backlog.mem_limit 50M
  storage.max_chunks_up     10
  [INPUT]
  name                              pulsar
  tag                               syslog-192.168.0.4-1
  alias                             syslog-192.168.0.4-1
  endpoint                          pulsar://pulsar.pulsar.svc.cluster.local:6650
  tenant                            public
  namespace                         default
  topic                             syslog-192.168.0.4-1
  storage.type                      filesystem
  storage.pause_on_chunks_overlimit on
  batch                             10000000
  format                            json
  threaded                          on
  [OUTPUT]
  name        pulsar
  match       *
  endpoint    pulsar://pulsar.pulsar.svc.cluster.local:6650
  tenant      public
  namespace   default
  separate    off
  standard    on
  format      avro
  outputSetup ./pulsar_ops.yaml
  backup      on
  debug       off
  workers     1
  [OUTPUT]
  name  null
  alias syslog-192.168.0.4-1
  match syslog-192.168.0.4-1

• Environment name and version (e.g. Kubernetes? What version?): Kubernetes
• Server type and version: x86 adm64
• Operating System and version: centos8
• Filters and plugins:

  [FILTER]
  name  modify
  set   foo bar
  set   standard test_log
  set   tenant 10000001
  match syslog-192.168.0.4-1
  alias syslog-192.168.0.4-1_2

[drbugfinder-work]

Which pulsar input/output plugin are you using? I cannot see any built-in plugin named like this. Maybe there's an issue in a custom plugin

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[littlejoyo]

yes，pulsar plugin build by fluent-bit/src/proxy/go，input and output golang plugin

[thipokch]

I’m having the same issue with OpenTelemetry too

[drbugfinder-work]

As this is no official Fluent Bit plugin (at least to my knowledge), I don't see a reason why this issue was opened here. My first guess would be that there's a memleak in the implementation of that 3rd party plugin.

In case it really is an issue with the Fluent Bit golang interface, I think the best way to narrow it down would be to create a very simple dummy golang plugin as a reproducer.

@patrick-stephens What's your opinion about this?

[patrick-stephens]

Yeah I think it needs a reproducer that is simple and using something officially supported by OSS. We can't be debugging every possible implementation. It may be there is some general issue with the golang interface but then it should be reproducible.

We need a reproducible problem report with the provided plugins or a way to do so. I'm not sure if @thipokch is saying there is a problem using the official OTEL input or output plugin either.

[thipokch]

I’m using OTEL as output.

P.S. I’m not sure if this is relevant, but it might be useful. In Grafana Alloy, they implemented memory limiter for OTEL:

https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.memory_limiter/

https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.