search

awslabs / aws-fluent-plugin-kinesis

Amazon Kinesis output plugin for Fluentd
Apache License 2.0
293 stars 96 forks source link

fluentd does not respect flush_interval #179

Closed kvitali closed 5 years ago

kvitali commented 5 years ago

My fluentd kinesis plugin version is 3.0.0. My config file section is:

`<match kinesis.syslog.**> @type copy

@type stdout @type kinesis_streams region "us-east-1" stream_name "audit" flush_interval 5s num_threads 4

`

The plugin does not respect the 5s flush interval. Sometimes it flushes 40 or 50 secs after actual event was produced. I learned this by comparing the timestamp fetched in my custom output code (for application purposes) vs 'time' field that is pushed to Kinesis stream.

simukappu commented 5 years ago

Hi @kvitali,

Which version of Fluentd are you using? Do you write flush_interval config in buffer section? We have updated README to find this easier.

kvitali commented 5 years ago

Hi @simukappu ,

We use Fluentd 1.2. As you can see in config I mentioned above, we don't write flush_interval in buffer section, we put it in plugin section. However, this works perfectly with Fluentd 1.2 + Kinesis plugin 2.1.1. So, the change must happened on Kinesis plugin side in 3.0.0 version.

simukappu commented 5 years ago

Thank you for sharing your environment. These buffer section configurations are Fluentd configuration, not implemented in this plugin. Please see this Fluentd document. Also, we have no change about these configurations between plugin v2.1.1 and v3.0.0. See this comparison. Can you try to put configurations in buffer section? Or, can you confirm config and results with previous version?

kvitali commented 5 years ago

thanks checking this proposal

kvitali commented 5 years ago

Hi @simukappu ,

After I moved the params under buffer section it works ok for plugin 3.1.0 version. Also works fine with fluentd 1.3

thanks