search

fluent-plugins-nursery / fluent-plugin-systemd

This is a fluentd input plugin. It reads logs from the systemd journal.
Apache License 2.0
153 stars 43 forks source link

Plug a memory leaks on every reload #91

Closed kenhys closed 3 years ago

kenhys commented 3 years ago

The resources must be freed during shutdown process when fluentd graceful reload is executed. Without it, fluentd --under-supervisor process keeps to consume memory.

Reproducible steps:

  1. Use the following fluent.conf:
<system>
  log_level info
  rpc_endpoint "127.0.0.1:24444"
</system>

<source>
  @type sample
  auto_increment_key count
  tag sample
</source>

<match sample>
  @type stdout
</match>Before:
  1. Launch fluentd -c fluent.conf
  2. Execute curl http://127.0.0.1:24444/api/config.gracefulReload few times. less than 10 times is enough.

FYI: https://github.com/fluent/fluentd/issues/3342

Before:

Every fluentd is reloaded, the value of top (RES) increases.

  % while true; do top -b -n 1|\grep ruby;sleep 1;done
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1247164  82512  47536 S   0.0   0.3   0:00.48 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1247324  82776  47536 S   0.0   0.3   0:00.49 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1777844  99824  64484 S   0.0   0.3   0:00.49 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1777844  99824  64484 S   0.0   0.3   0:00.49 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.53 ruby
  126506 kenhys    20   0 1777976 100072  64484 S   0.0   0.3   0:00.50 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.53 ruby
  126506 kenhys    20   0 2308496 119016  83348 S   0.0   0.4   0:00.51 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.54 ruby
  126506 kenhys    20   0 2308496 119016  83348 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.54 ruby
  126506 kenhys    20   0 2839016 138284 102556 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.54 ruby
  126506 kenhys    20   0 2839016 138284 102556 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.55 ruby
  126506 kenhys    20   0 2839016 138356 102620 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.55 ruby
  126506 kenhys    20   0 3369536 155020 119256 S   0.0   0.5   0:00.53 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.55 ruby
  126506 kenhys    20   0 3369536 155020 119256 S   0.0   0.5   0:00.54 ruby
  126501 kenhys    20   0  193052  46664  11068 S   6.7   0.1   0:00.56 ruby
  126506 kenhys    20   0 3900056 173852 138016 S   0.0   0.5   0:00.54 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 3900056 173852 138016 S   0.0   0.5   0:00.54 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 4430576 193100 157192 S   0.0   0.6   0:00.54 ruby
  126501 kenhys    20   0  193052  46780  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 4430576 193104 157192 S   0.0   0.6   0:00.54 ruby
  126501 kenhys    20   0  193052  46780  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 4430576 193152 157192 S   0.0   0.6   0:00.54 ruby

After:

Even though fluentd is reloaded, the value of RES does not increase.

  % while true; do top -b -n 1|\grep ruby;sleep 1;done
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.59 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.59 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.59 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  193016  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  186144  45892  11244 S   0.0   0.1   0:00.60 ruby
  117844 kenhys    20   0  193016  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  716664  63160  28372 S   0.0   0.2   0:00.60 ruby
  117844 kenhys    20   0  193016  46388  10912 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  716664  63216  28436 S   0.0   0.2   0:00.60 ruby
cosmo0920 commented 3 years ago

@errm Could you kindly take a look?

ashie commented 3 years ago

@errm ping

kenhys commented 3 years ago

Thanks!

errm commented 3 years ago

Thanks for the fix @kenhys I just pushed 1.0.5 to rubygems

ashie commented 3 years ago

Thanks!