search

fluent / fluent-operator

Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Apache License 2.0
587 stars 250 forks source link

How to add new plugins in the fluentd Output #254

Open henrikrexed opened 2 years ago

henrikrexed commented 2 years ago

Is your feature request related to a problem? Please describe.

it seems that adding new plugins requires to

This constraint will slow the adoption , because lots of customers are using community plugins

Describe the solution you'd like

Provide a guide helping our community in building custom images.

To avoid modifying the various CRDs, we should have a way of adding "custom" Filter, output, parser steps. modifying

Describe alternatives you've considered

For the moment my only workaround is to use the operator to manage fluentbit, and then deploy fluentd as a daemonset to be able to use the custom plugins.

Additional context

No response

github-actions[bot] commented 2 years ago

@henrikrexed this issue was automatically closed because it did not follow the issue template

zhu733756 commented 2 years ago

Thanks for the great ideas.

For the moment my only workaround is to use the operator to manage fluentbit, and then deploy fluentd as a daemonset to be able to use the custom plugins.

Our design is to deploy fluentbit as a daemonset to collect logs, sink to the storages like es, loki, etc. Fluentd acts as a statefulset workload to receieve the logs from fluentbit or apps, also could sink to the storages.

benjaminhuo commented 2 years ago

@henrikrexed I agree with the idea of Provide a guide helping our community in building custom images.

Regarding the second point to avoid modifying the various CRDs, we should have a way of adding "custom" Filter, output, parser steps, there're two methods to support various plugins:

The annotation method is flexible but there is no validation for it. The second is not that flexible, but there is validation for it.

We choose the second method because of the validation it provides and some fluentbit plugins are added by community contributors.

For fluentd, we'll consider the annotation method to allow more flexibility.

And we don't recommend using Fluentd as a daemonset because it's fluentbit's role（acting as a logging agent）.

Fluentd receives logs from fluentbit or applications through the network and does more advanced processing.

@wanjunlei @wenchajun @zhu733756

henrikrexed commented 2 years ago

I agree but in my use case I only wanted to use Fluentd to forward the logs to storage solution. Unfortunately, there are no fluentbit plugin for this solution, that is the reason why I must use fluentd.

So, in the case of a custom output plugin of fluentd , the user needs to create a custom image , adding the section of the pipeline calling the output plugin in the fluentd.conf file?

Thanks,

From: Benjamin Huo @.> Sent: lundi 28 mars 2022 10:58 To: fluent/fluent-operator @.> Cc: Rexed, Henrik @.>; Mention @.> Subject: Re: [fluent/fluent-operator] How to add new plugins in the fluentd Output (Issue #254)

@henrikrexedhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhenrikrexed&data=04%7C01%7Chenrik.rexed%40dynatrace.com%7C03314cda30344b2d562508da10991bcc%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637840547076001502%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=HDlQAJrhLYs9Jd31JWrA8mE3ceptgLYTxZ%2FhUU4jL4U%3D&reserved=0 I agree with the idea of Provide a guide helping our community in building custom images.

Regarding the second point to avoid modifying the various CRDs, we should have a way of adding "custom" Filter, output, parser steps, there're two methods to support various plugins:

The annotation method is flexible but there is no validation for it. The second is not that flexible, but there is validation for it.

We choose the second method because of the validation it provides and some fluentbit plugins are added by community contributed.

For fluentd, we'll consider the annotation method to allow more flexibility.

And we don't recommend using Fluentd as a daemonset because it's fluentbit's role（acting as a logging agent）.

Fluentd receives logs from fluentbit or applications through network and do more advanced process.

- Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffluent%2Ffluent-operator%2Fissues%2F254%23issuecomment-1080377508&data=04%7C01%7Chenrik.rexed%40dynatrace.com%7C03314cda30344b2d562508da10991bcc%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637840547076001502%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=nVBS0hyDLeGr1VoBlyGwAUxKhiOMVSpUBMWVxWnLJXQ%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATQC5U4JI63QQDVOGEXNGM3VCFYCVANCNFSM5RUUBCPA&data=04%7C01%7Chenrik.rexed%40dynatrace.com%7C03314cda30344b2d562508da10991bcc%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637840547076001502%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2FumQ55p2SoQL2xDd2Dqa5exiqc6D0kz5q7li8O1GLw%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

petergvizd commented 2 years ago

Hello @benjaminhuo

we would like to use fluent-operator with fluent-plugin-grok-parser, so I would like to ask, what is meant by "add plugin settings as the current method"? Do you mean by this, to add support for plugin to code, CRDs and image or something else?

wenchajun commented 2 years ago

Hello @benjaminhuo

we would like to use fluent-operator with fluent-plugin-grok-parser, so I would like to ask, what is meant by "add plugin settings as the current method"? Do you mean by this, to add support for plugin to code, CRDs and image or something else?

Hi, you can refer to these use cases https://github.com/fluent/fluent-operator/tree/master/apis/fluentbit/v1alpha2/plugins

benjaminhuo commented 2 years ago

There're some discussions about adding custom plugins in https://github.com/fluent/fluent-operator/issues/301#issuecomment-1152362843 We'll see if that is a feasible approach.