aws-samples / amazon-cloudwatch-container-insights

CloudWatch Agent Dockerfile and K8s YAML templates for CloudWatch Container Insights.
MIT No Attribution
165 stars 108 forks source link

Reapply "reduce default log level from info to error to reduce unnece… #157

Closed chadpatel closed 11 months ago

chadpatel commented 11 months ago

…ssary logs being published (#155)" (#156)

This reverts commit 4e4d41ad9501dcce315845606a5ff1e2dd2d8976. and re-applies https://github.com/aws-samples/amazon-cloudwatch-container-insights/pull/155

Description of the issue

Fluent bit default log level is set to info, which is pretty chatty. This is causing customers to pay unnecessarily large log bills, particularly when their logs are not in the "kubernetes" format which results in this log message:

[2022/06/30 06:09:29] [ warn] [record accessor] translation failed, root key=kubernetes

https://docs.fluentbit.io/manual/v/1.9-pre/pipeline/outputs/cloudwatch

If the kubernetes structure is not found in the log record, then the log_group_name and log_stream_prefix will be used instead, and Fluent Bit will log an error like:
...

Other customers have reported a large amount of this unnecessary warning

2023-09-21T20:14:51.860592207Z stderr F [2023/09/21 20:14:51] [ warn] [parser:_ml_cri] invalid time format %Y-%m-%dT%H:%M:%S.%L%z for '2023-09-21T20:14:51.860487043Z stderr F [2023/09/21 20:14:51] [ warn] [parser:_ml_cri] invalid time format %Y-%m-%dT%H:%M:%S.%L%z for '2023-09-21T20:14:51.860372704Z'

Description of changes

Change the default log level in our samples from info to error

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

I spun up a new cluster and deployed the enhanced CI yaml I also verified info level application logs are still uploaded to cloudwatch

Requirements

Before committing the code, please verify the following: