Currently input plugins run on a schedule and are called once per second. This is not ideal for dealing with streaming / event driven data, as that data may reach the input plugin at an uneven rate. Events should ideally be collected as soon as they are available (I believe this is how other plugins work, like the tail plugin).
With the current implementation, the fastest rate data chunks could be consumed would be every second.
I believe this would require a wrapper of flb_input_set_collector_event in the plugin API, however from what I can tell, Fluent Bit core would probably need to be updated as well since the proxy plugin which is used to wrap Go plugins is responsible for setting this timer, i.e. by calling flb_input_set_collector_time (here: https://github.com/fluent/fluent-bit/blob/master/src/flb_plugin_proxy.c#L151-L155).
Currently input plugins run on a schedule and are called once per second. This is not ideal for dealing with streaming / event driven data, as that data may reach the input plugin at an uneven rate. Events should ideally be collected as soon as they are available (I believe this is how other plugins work, like the
tailplugin).With the current implementation, the fastest rate data chunks could be consumed would be every second.
I believe this would require a wrapper of
flb_input_set_collector_eventin the plugin API, however from what I can tell, Fluent Bit core would probably need to be updated as well since the proxy plugin which is used to wrap Go plugins is responsible for setting this timer, i.e. by callingflb_input_set_collector_time(here: https://github.com/fluent/fluent-bit/blob/master/src/flb_plugin_proxy.c#L151-L155).