Configure monitoring for Registry Sweeper ECS cluster

[sjoshi-jpl]

Setup health monitors to monitor the CPU / Memory Utilization for each ECS task for registry-sweeper service

[sjoshi-jpl]

Currently working on setting up EventBridge to schedule up a task for each domain targeting prod. Will need PROV_Endpoint var value to move forward for each domain.

[sjoshi-jpl]

@tloubrieu-jpl @jordanpadams

What's completed?

• 10 new ECS Provenance task definitions, with 10 new schedules
• ECS Registry-Sweeper Monitoring Dashboard (All ECS Registry-Sweeper Domains tasks). Currently we're monitoring for CPU and Memory Utilization for each task. Also, TaskCount at Cluster Level.
• 4 Alarms (CPU, Memory, TaskCount) setup for Provenance Monitoring (1 is a composite alarm that goes off if either of the remaining 3 alarms go off).
• ECS Task Remediations (#55)
• Documentation for all ECS / OpenSearch related configurations.

What's pending (first decide if it's required)?

• Do we really need a new Registry-Sweeper Cluster? If so we'll need to switch all of the above tasks to point to that new cluster and also have to create new Alarm. Effort wise that's 1 full day worth of effort including testing.
• Do we need to setup monitoring and task definitions for EN-DELTA and ATM-DELTA clusters?
• Do we need to setup alarms for each OpenSearch Cluster? If so which metrics should we monitor? Currently we have a FreeStorageSpace alarm already setup. If we create more alarms for more metrics, then we might need to create a composite alarm for each to reduce noise.
• Do we need to setup Cluster level alarms and task level alarms separately for each ECS cluster?

I am happy to give you guys a demo of what's already done and then we can decide on the above questions.

[tloubrieu-jpl]

Thanks @sjoshi-jpl , a demo sounds like a good idea for the up coming sprint review, this Thursday.

[sjoshi-jpl]

@tloubrieu-jpl as discussed over teams chat with @jordanpadams we can leave what we have for now and then if the cluster level alarms for provenance are not actively helping us manage the task level resources, we can work on creating CPU/Memory alarms at the task level for provenance.

For now we have the essential alarms covered and it might not be worth to turn on container insights for every Registry-API cluster, especially if we are planning to move towards multi-tenancy. We can make more robust monitoring once we're using multi-tenancy because at that point we won't have multiple clusters / tasks to manage.

Moving this ticket under review. I'll demo what I have configured so far on Thursday.

[jordanpadams]

An initial pass at this has been completed and demoed at last sprint review on 8/10