search

weaveworks / eks-quickstart-app-dev

Example flux manifests for eksctl gitops
Other
56 stars 43 forks source link

Investigate the possibility to use fluentbit instead of fluentd #21

Closed gemagomez closed 5 years ago

gemagomez commented 5 years ago

Overview

Let's investigate whether it makes sense to change fluentd for fluentbit[1] and proceed accordingly.

[1] https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/

cPu1 commented 5 years ago

Fluent Bit's deployment is similar to Fluentd's in that it requires a ClusterRole for accessing pods and some logging configuration. Access to CloudWatch logging is also covered by enabling the cloudWatch addon policy in the ClusterConfig. It's much more resource-efficient than Fluentd as stated in the benchmarks in the linked blog post. However, deploying the exact setup linked in the blog post (streaming via Amazon Kinesis Data Firehose) would be too specific and opinionated for a quickstart component, and would also require manually creating IAM policies (as opposed to just specifying them in iam.withAddonPolicies in the ClusterConfig).

A good reason to continue using Fluentd is its much larger plugin ecosystem. Some users may already be using Fluentd with plugins that are unavailable for Fluent Bit. See https://github.com/aws/containers-roadmap/issues/10#issuecomment-510753911 and https://github.com/aws/containers-roadmap/issues/10#issuecomment-520683498

cPu1 commented 5 years ago

An issue has been opened to either replace Fluentd with Fluent Bit, or use them in tandem: https://github.com/weaveworks/eks-quickstart-app-dev/issues/28