aws-observability / observability-best-practices

Observability best practices on AWS
https://aws-observability.github.io/observability-best-practices/
MIT No Attribution
224 stars 69 forks source link

Cost savings advice for Container Insights maybe outdated #124

Open tgraupne opened 6 months ago

tgraupne commented 6 months ago

While testing AWS CloudWatch Container Insights with Enhanced Observability I felt the urge to provide a configuration with a better fit to our needs. Reading through your guides, the section about "Cost savings with Container Insights on Amazon EKS" were very interesting. In particular the subsection called: "Customize Metrics and Dimensions".

https://github.com/aws-observability/observability-best-practices/blob/97774e64728c09fc85fb37b1f618af8d722915cb/docs/en/guides/containers/aws-native/eks/amazon-cloudwatch-container-insights.md?plain=1#L158

The solution presented there seems easy to understand and very straight forward. However, nowadays, after the release of CloudWatch Container Insights with Enhanced Observability, it seems to me the presented guides here are out of date.

Why? Because I believe if one follows the suggested installation method (EKS add-on), the documented otel-agent-conf is not used. But, looking at the log of the deployed cloudwatch-agent, it still uses the awsemfexporter. Hence, I believe it should be possible to use the described metric_declaraiont:

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/5ccdbe08c6a2a43b7c6c7f9c0031a4b0348394a9/exporter/awsemfexporter/README.md#metric_declaration

I would be very thankful if someone could help me understand how we could move forward here.