[Datadog] Eval/Setup RED Monitoring for CI/CD:ArgoCD

[ph-One]

Description

As a platform engineer Datadog monitors and alerts need to be evaluated and/or setup for CI/CD:ArgoCD

Acceptance Criteria

• [ ] CI/CD:ArgoCD is evaluated for monitoring needs
• [ ] Monitors are created, if appropriate
• [ ] Alerts are created, if appropriate

Notes

• This has a wide audience, from PO's to Dev, and everyone in between
• Remember to keep the scope to Argo and not each individual application Argo is deploying
• Take a look at the ArgoCD UI to better understand the application
• The RED method for services
  • Rate: The number of requests the service is handling per second
  • Error: The number of failed requests per second
  • Duration: The amount of time each request takes (latency)

[hgbarreto]

Rate - Redis Request Count (WIP) Error - App Degraded, App OutOfSync, K8s Bad Responses Duration - Redis Request Duration (WIP)

[hgbarreto]

Before adding the Redis Request monitors to the devops alerts, I would like them to be reviewed. Unsure if it is configured in a useful way.

[npeterson54]

Not quite complete, rolling into next sprint

[ph-One]

Platform - ArgoCD: Redis Request Duration High (https://vagov.ddog-gov.com/monitors/116864):

• [x] Help us better understand the metric and what it's telling us

[npeterson54]

We were not able to complete this due to security issues taking time of devs. Moving into next sprint.

[hgbarreto]

Duration - Redis Request Duration Figured out how to convert into milliseconds. Please review and let me know.