create RFC 0004 for fluent-bit

[chanwit]

Closes https://github.com/weaveworks/weave-gitops/issues/3195

This RFC proposes the integration of Fluent-Bit into Weave GitOps as a log forwarder to collect logs from all pods in a VCluster. Fluent-Bit is written in C/C++ and is lightweight compared to FluentD, which is written in Ruby.

It will be installed using Helm and configured to read logs from all containers in the /var/log/containers/ directory, apply the kubernetes filter to merge and exclude relevant log lines, and output the logs to an S3 bucket. An API will also be created to allow users to retrieve logs by pod name and namespace. The storage type and log retention policy for the S3 bucket will be considered, and appropriate documentation will be created for other users and maintainers.

[makkes]

A few thoughts that came up when I read the RFC:

• We need to clarify the implications of making fluent-bit part of the Weave GitOps product suite, in particular:
  • if we ship fluent-bit as part of the product we will be responsible for supporting it, too.
  • do we envision encouraging customers to use that fluent-bit instance for other purposes than what we foresee (mainly GitOps Run at the moment)?
  • so far Weave GitOps has been a very non-opinionated product being usable in all kinds of different environments and with all different kinds of components installed on clusters. This would change if we shipped fluent-bit or any other component that we do not build ourselves. We could ask ourselves, why don't we ship kube-prometheus-stack, dex or cert-manager?
• From all I understand (please correct me if I'm wrong, @chanwit) this RFC came out of the necessity to gather logs of components installed through GitOps Run but the RFC doesn't mention any alternatives to installing fluent-bit.
• If we want to make use of fluent-bit for the GitOps Run use case, we should clearly point that out in the RFC and tighten the scope of it to that specific use case.

[chanwit]

@makkes Thank you for the review. Let's discuss with @JamWils about the scope of this log aggregation mechanism. That's the product question.

if we ship fluent-bit as part of the product we will be responsible for supporting it, too.

We integrated vcluster as one of our components. We also use many libraries as part of the OSS. That's a fact for now.

[JamWils]

if we ship fluent-bit as part of the product we will be responsible for supporting it, too.

No, this does not mean we support fluent-bit. This is a technical solution to do log aggregation in gitops run. It is enabling a product feature similar to vCluster and React. I don't think we support fluent-bit, but we support that the logs aggregate appropriately.

Putting my engineering manager hat on, if we get a quick win of aggregating logs with fluent-bit and don't have to write our own aggregator then I think that is a huge win. From a product perspective, we need a dynamic log system. If there are multiple apps then we want to show the logs for all of those, we also want to show the logs of the various controllers we have deployed, etc. If fluent-bit can do this (I believe it can) then we should consider adopting this technology.

I hope that helps, if we need to set up a meeting to discuss then I am happy to.

If we want to make use of fluent-bit for the GitOps Run use case, we should clearly point that out in the RFC and tighten the scope of it to that specific use case.

Agreed!

[chanwit]

If fluent-bit can do this (I believe it can) then we should consider adopting this technology.

According to the spike outcomes, yes it can do that.

If we want to make use of fluent-bit for the GitOps Run use case, we should clearly point that out in the RFC and tighten the scope of it to that specific use case.

Agreed!

Acked. I'll scope this RFC down to the use case.

[enekofb]

@makkes @chanwit this seems a good candidate to share with tech leads in our tuesday catchup?

[makkes]

@makkes @chanwit this seems a good candidate to share with tech leads in our tuesday catchup?

I added an item to the backlog at https://www.notion.so/weaveworks/7f5bdf188d0a4a84a583935bbce69a8a?v=091591e0b8f04b03bd4243197e22c6eb.

[chanwit]

Hi all,

I got this RFC updated. Please let me know if you have any comments.