[ECS] [Agent]: publish ECS Agent Status Metrics to CloudWatch Metrics

[colinbjohnson]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

I would like ECS Agent Connected status published to CloudWatch Metrics.

Which service(s) is this request for?

ECS with EC2.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

The typical use case would be to alert on systems where the ECS Agent on a given Container Instance has been disconnected for a period of time and to respond to this event (either through a manual or automated means). This is difficult because:

• utilizing "event based" monitoring/alerting is difficult because the agent connects/disconnects "several times per house during normal operations" (in other words, if you alert on each disconnect that will be an issue)
• the telemetry data we would like to utilize is not available

Currently:

Are you currently working around this issue?

• I have written a lambda function (about 40 lines of code) that publishes a metric to a service to allow alerting (this service could be CloudWatch)

Additional context

• I believe very few customers know this is an issue or they encounter the issue infrequently enough they accept the increase cost resulting from running disconnected Container Instances
• AWS has published (Monitoring the Amazon ECS Agent)[https://aws.amazon.com/blogs/containers/monitoring-the-amazon-ecs-agent/] which provides evidence that AWS is aware of an issue ("Platform operators often look for guidance on how to monitor the availability of the ECS Agent, and therefore the ECS Container Instance, by ensuring the failure relevant alerts are in place"). There are reasons why an ECS customer would not want this solution:
  • an expectation that this (monitoring agent status) is an important feature of the ECS platform and should be provided/built-in
  • perception of lack of adoption (in 2+ years this repository has three stars and three forks) and almost no public Internet links
  • complexity: the resource count: 11 and the variety of different resource types places a cognitive load/knowledge requirements on the user/implementer where a more simply CloudWatch Metric and Alert could accomplish the similar task easily
  • the status (it's in "aws-samples" implying it isn't fully supported)

Attachments

I would, happily, provide a link to the lambda solution I've written to demonstrate both:

1. How I believe the metrics should be produced (essentially, I copied the EC2 Instance Status metrics)
2. Provide a jumping off point for discussions about how to implement this

[Nevon]

I encountered and worked around the exact same thing just a few weeks ago. Originally I implemented the solution outlined in the AWS article but I found it to cause endless amounts of what amounts to false positives due to how it is designed. The design is not checking that a container instance remains disconnected for X minutes. It is only checking that a container instance was disconnected at minute 0 and then also at minute X. Given that connectivity can fluctuate, over a large enough fleet of instances, it happens fairly frequently that a container instance is disconnected, then connects again, then X minutes later disconnects again and causes an alert.

The improved design we implemented still sends ECS Container Instance State Change events to an SQS queue, but instead of delaying the processing of the messages for X minutes and then checking the status once, we instead simply check the container instance status from the message handler, and if the instance is still disconnected we emit a metric and put the message back on the queue to be processed again a minute later. That way we get a steady stream of metrics with about a minute's granularity, allowing for monitors that alert when an instance is continuously disconnected for more than X minutes.

Despite having this workaround in place, I too think that this should be emitted as a Cloudwatch metric out of the box.