The throttler filter never removes inactive groups from the counter hash. This fix implements LRU eviction to remove inactive groups in order to relieve memory usage. Eviction is attempted every time a new event arrives so that the hash can decrease in size and not increase indefinitely.
Testing Performed:
Fluentd was configured to use the fluent-plugin-throttle filter and read json logs from a file.
Throttler before fix
1,000,000 unique logs
cpu time: 00:27.40
RSS: 429.4 MB
2,000,000 unique logs
cpu time: 01:08.36
RSS: 784.3 MB
Throttler after fix
1,000,000 unique logs
cpu time: 00:35.95
RSS: 160.1 MB
2,000,000 unique logs
cpu time: 01:11.96
RSS: 161.5 MB
LRU eviction was tested and verified through the debug logs when running fluentd on verbose mode.
Description
The throttler filter never removes inactive groups from the counter hash. This fix implements LRU eviction to remove inactive groups in order to relieve memory usage. Eviction is attempted every time a new event arrives so that the hash can decrease in size and not increase indefinitely.
Testing Performed:
Fluentd was configured to use the fluent-plugin-throttle filter and read json logs from a file.
Throttler before fix
1,000,000 unique logs cpu time: 00:27.40 RSS: 429.4 MB
2,000,000 unique logs cpu time: 01:08.36 RSS: 784.3 MB
Throttler after fix
1,000,000 unique logs cpu time: 00:35.95 RSS: 160.1 MB
2,000,000 unique logs cpu time: 01:11.96 RSS: 161.5 MB
LRU eviction was tested and verified through the debug logs when running fluentd on verbose mode.