[EKS] [request]: Customize Cloudwatch Observability AddOn

[if-jeremy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build?

Allow more customization options to the Cloudwatch Observability addon

Which service(s) is this request for? This could be Fargate, ECS, EKS, ECR

EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We would like to be able to customize the FluentBit logging portion of the Observability addon. Currently, to my knowledge, there is no way to customize the fluent-bit-config ConfigMap, short of using kubectl patch, which can be problematic when doing IaC. Just as an example, we would like to be able to customize log_group_name, log_stream_name, auto_create_group, and log_retention_days parameters.

Are you currently working around this issue? How are you currently solving this problem?

Looking into using kubectl patch to modify the ConfigMap, but that's not a real clean solution for IaC

Additional context Anything else we should know?

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[mikestef9]

CloudWatch observability addon is meant to be opinionated and power the enhanced observability experience for EKS. Customizing log group names likely breaks that dashboard and experience.

If you just want an addon to customize logging for your cluster, ADOT is supported, and support for logs is now GA.

[zensai3805]

I agree with the potential issues that could arise from editing log group names. However, I am concerned about the costs associated with sending all logs to CloudWatch Logs. I would like to suggest the following customizations:

• Collect logs only from specific applications.
• Instead of sending all logs, allow specifying a sampling rate to send only a portion of the logs.

[zensai3805]

With the following change, it seems possible to modify the FluentBit ConfigMap: https://github.com/aws-observability/helm-charts/pull/43

Since this EKS addon uses this Chart, we have confirmed that it is possible to edit the configuration using configuration_values when installing the addon with Terraform.

While it is unclear if this change aligns with AWS's addon policy, will this be maintained in the future?

[sky333999]

To provide the flexibility around the volume of logs emitted, we do now support allowing customization of the FluentBit config via the add-on advanced config. Please see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Observability-EKS-addon.html#CloudWatch-Observability-EKS-addon-CustomFluentBit for further details.

As noted, changing the defaults should be done carefully to avoid breaking existing integrations with solutions such as Container Insights.