search

fujiwara / fluent-plugin-suppress

fluentd plugin to suppress same messages.
Other
17 stars 13 forks source link

Two duplicate records are found even after suppression-Intermittent issue #14

Open rayalaka opened 4 years ago

rayalaka commented 4 years ago

Sometimes, When similar multiple records are pushed to fluentd(assume 1000 records in a loop), two duplicate records are still visible in the fluentd logs instead of 1. Please note that there is no timestamp difference identified between these two records.

Is this a race condition? filter_suppress.rb does not have a def multi_workers_ready setting to true. But, the same is present in out_suppress.rb. Do we need to add a method multi_workers_ready in out_suppress.rb as well? If this not a race condition then let me know if I am missing something. The filter conf is as below:

<filter **>
 @type suppress
 interval 300
 num 1
 attr_keys hostname,message
</filter>

Please help me to get this resolved.

HelenBaoFinnAI commented 5 days ago

I have the same issue, any updates on this issue or suggestions?