Open steveellis opened 2 years ago
Ok actually this is what we should be doing to solve this: https://github.com/aws-observability/aws-otel-helm-charts/tree/master/charts/adot-exporter-for-eks-on-ec2. This way we can deploy the agent with helm (and pulumi) and not have to use the shell script or the yamls at all.
This can also be used to deploy fluent bit.
What ADOT is and how it fits into the CloudWatch agent: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-otel.html.
Ref #252 and #206.
Ok configuring the chart it is possible to send data only to CW but it seems like AMP/Graphana might be easy to set up as well: https://github.com/aws-observability/aws-otel-helm-charts/blob/master/charts/adot-exporter-for-eks-on-ec2/documentation/metrics_to_cloudwatch_amp.md#sending-metrics-and-logs-to-cloudwatch-container-insights-only.
I think I'm going to try to get all of this going at once, metrics and logging using this helm chart, under this ticket.
There's an issue with the aws helm chart. They've added some images to the chart docs and need to add a .helmignore
file to ignore them. Since the chart is unusable and they say they are supporting it, I'd expect it to be fixed quickly: https://github.com/aws-observability/aws-otel-helm-charts/issues/30.
Metrics and logging are working on the scratch instance when deployed via script. The helm chart isn't working yet, the ADOT collector is however for both logging and metrics. Closes #174.
Log retention should be part of this PR
cc @jwfuller
Log retention appears to be handled by this variable in the ConfigMap for FluentBit. If we deploy via kubectl apply
(not the helm chart) this variable will need to be added since it isn't present by default.
That's all for retention of logs. Retention of metrics is another matter and it's not immediately clear where that would get controlled. It does seem like it should be a variable somewhere in the cwagentconfig.json
here.
What options are there for working with these aws yaml files in pulumi if we can't get the helm chart to work? Getting the helm chart working is being tracked here.
Only 2 would let you neatly handle passing parameters to the yaml (the cluster name for example is a runtime value).
If we apply the yaml in 1 it seems like we can change values inside that yaml using transformations.
It does seem like it should be a variable somewhere in the cwagentconfig.json
Maybe in these lines? The output conf section looks similar to in the ConfigMap for FluentBit
@jwfuller I think your first instinct was right about this. The cloudwatch agent container image referred to in the daemonset file is more than 2 years old. This agent code has been under active development.
https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/d60789889e0b679947e73d393618f6385a2a64ec/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-daemonset.yaml#L18
Think this should be
latest
.I think we should indeed bring these yamls into our repo.