rubrikinc / fluent-plugin-throttle

Rate limiting for fluentd
Apache License 2.0
41 stars 24 forks source link

Throttling is not resetting after setting the group_reset_rate_s #21

Closed citrocity closed 1 year ago

citrocity commented 1 year ago

Hi,

I know this may be a long shot as I haven't seen any activity on this repository, but I was playing around with this plugin and even though it does throttle the logs to 1000, it doesn't seem to reset after 15s. I don't receive any logs after the first 1000. The output is elasticsearch.

<filter throttle.**>
  @type throttle
  group_key random.env
  group_bucket_period_s  60
  group_bucket_limit     1000
  group_drop_logs        true
  group_reset_rate_s     15
</filter>

Below are the versions I am using:

ruby version 2.7.0
fluent-config-regexp-type (1.0.0)
fluent-plugin-beats (1.1.0)
fluent-plugin-elasticsearch (5.2.3)
fluent-plugin-filter-list (0.7.5)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.1)
fluent-plugin-throttle (0.0.5)
fluentd (1.15.2)

The logs show it being throttle (note the group shows nil so i guess it isnt resolving but still works as i can see 1000 logs)

2022-12-16 14:25:38 +0000 [warn]: #0 rate exceeded group_key=[nil] rate_s=286 period_s=60 limit=1000 rate_limit_s=16 reset_rate_s=15
2022-12-16 14:25:38.418280984 +0000 fluent.warn: {"group_key":[null],"rate_s":286,"period_s":60,"limit":1000,"rate_limit_s":16,"reset_rate_s":15,"message":"rate exceeded group_key=[nil] rate_s=286 period_s=60 limit=1000 rate_limit_s=16 reset_rate_s=15"}

Any help would be appreciated