vmware / fluent-plugin-vmware-loginsight

Fluentd plugin to forward logs to VMware Aria Operations for Logs
Other
10 stars 14 forks source link

Add support for multi process workers #26

Closed yajith closed 1 year ago

yajith commented 2 years ago

Is your feature request related to a problem? Please describe.

At present we have a basic configuration where each log-forwarding server has one worker(default) but with the amount of log volume we are seeing the fluentd buffer queue length increasing and not being able to handle every log event quickly.

Describe the solution you'd like

To address above issue, we would like to make use of the multi-process worker feature in fluentd to better utilize our log-forwarding servers. However we've noticed that this feature is not supported by the plugin and encountering following error. "Plugin 'vmware_loginsight' does not support multi workers configuration

https://docs.fluentd.org/deployment/multi-process-workers

Describe alternatives you've considered

We have considered setting up more log forwarding servers in each location and re-configure source systems to send logs to a selected log-forwarder system. Due to the complexity this introduces to our current setup, further investigation is still in progress to identify if there are better alternative solutions.

Additional context

We use td-agent distribution of fluentd (as of now td-agent-4.3.0-1.el8.x86_64)

yajith commented 2 years ago

added #multi_workers_ready? method that returns true as explained in https://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released under pull request #27

mohitevishal commented 1 year ago

Thanks for your contribution. The change as been merged https://github.com/vmware/fluent-plugin-vmware-loginsight/commit/1f3e3395aac167f8702329fb36a0385d7e955269