Re-write plugin to be compatible with current version of fluent-bit daemonset

[guineveresaenger]

Blocked by https://github.com/samsung-cnct/k2-logging-fluent-bit-daemonset/issues/4 Possibly blocked by upstream issue: https://github.com/fluent/fluent-bit-go/issues/6

Background for current state of affairs here

[davidewatson]

We should reevaluate whether we really need our own kafka plugin. There is a Kafka REST Proxy plugin now: http://fluentbit.io/documentation/current/output/

Will it suffice? And if not, should we focus on getting our plugin upstream?

[alejandroEsc]

@davidewatson @yenicapotediaz has a few issues to bring up before i feel comfortable bringing our plugin tool upstream.

[davidewatson]

What are they? I am not advocating that we upstream our plugin. I want to understand why we need it in the first place.

[alejandroEsc]

@davidewatson good question, i thought we wanted control on how we parse the data into our own message type? I dont know. Anyway, those issues are to be written. Will mention here when they are. Also If we are keeping this project, may i suggest that the issues be expounded on a bit, i have trouble understanding what they issue is per se.

[leahnp]

To use the fluentbit-native Kafka REST Proxy plugin we would need to add the "Kafka REST Proxy" to our central-logging deployment, but it seems like a reasonable option.

I am working on the fluent-bit docker image right now for https://github.com/samsung-cnct/issues/issues/68 and the libraries for the kafka output plugin are causing some build issues and limiting the base image options we can use.

[coffeepac]

underlying issue has been closed, this is ready to go forward.

there is also a question if we want to do use this at all. for now, I would like to at least go through the steps to make sure that the work eduardo did to make the golang adapter layer functional works. after we have that we can evaluate if we want to use our own plugin and eventually upstream it or abandon ship and use the current upstream plugin.

[coffeepac]

@yenicapotediaz to add ToReview criteria.

[StevenACoffman]

You may be interested in comparing this to fluent/fluent-bit-kubernetes-logging#11 and fluent/fluent-bit#94

[coffeepac]

@StevenACoffman thanks for the heads ups! we will are not going to be using our own plugin and we have built an image and chart for running fluent-bit as a daemonset

I'm going to mark this as closed as we aren't using this plugin and we've already given a talk on how to write a golang plugin for fluentbit at kubecon so anything for this issue and probably this repo is now done.

[StevenACoffman]

@coffeepac Currently your image and chart still do use your own plugin, because fluent-bit has not fully implemented a pure kafka producer until fluent/fluent-bit#94 implements retry logic. That's a fine plan though for when it is.

Thanks for the update!