[Datadog] Eval/Setup RED Monitoring for EKS:Clusters

[ph-One]

Description

As a platform engineer Datadog monitors and alerts need to be evaluated and/or setup for EKS:Clusters

Acceptance Criteria

• [ ] EKS:Clusters is evaluated for monitoring needs
• [ ] Monitors are created, if appropriate
• [ ] Alerts are created, if appropriate

Notes

• kubelet
• kube-proxy
• coredns
• The RED method for services
  • Rate: The number of requests the service is handling per second
  • Error: The number of failed requests per second
  • Duration: The amount of time each request takes (latency)

[jhouse-solvd]

Note: This issue will carry into the next sprint. We've made significant progress, but there are several refinements needed to ensure comprehensive monitoring for EKS (Clusters & Ingress).

Also, we consulted w/ Platform Tech Team 1 today to inform them of the monitoring that's been put in place for EKS. They are migrating Vets-API to EKS and will be adding monitors for parts of the application that are not yet covered.

[it-harrison]

coredns

• rate monitor
• error rate monitor
• duration monitor

kubelet

• rate monitor
• error rate monitor
• request duration monitor Note: the values here do not make sense, i.e. they are far too high

[jhouse-solvd]

Re: Kube proxy, I'm not sure if this is needed, but you might take a look: https://github.com/DataDog/integrations-core/blob/master/kube_proxy/README.md

[ph-One]

https://vagov.ddog-gov.com/monitors/115057

• please add the "why" to the description
• revisit thresholds
• remove cluster name from the from
• understand why there's no data

[jhouse-solvd]

https://vagov.ddog-gov.com/monitors/115078

• understand what no error means
• filter out noerror if it doesn't mean an error

https://vagov.ddog-gov.com/monitors/115068

• add the "why" to the description
• remove the cluster name from the metrics
• explore different detection methods

https://vagov.ddog-gov.com/monitors/115081

• for kubelet operations rate, consider raising the thresholds
• add the "why" to the description

https://vagov.ddog-gov.com/monitors/115082

• add the "why" to the description

https://vagov.ddog-gov.com/monitors/115086

• add the "why" to the description
• check out Datadog documentation to better understand the metrics

[it-harrison]

• coredns rate updated. note: although data is pulled on the edit screen I'm not seeing anything after creating the monitor. still looking for a solution...
• coredns error updated. note: although data is pulled on the edit screen I'm not seeing anything after creating the monitor. still looking for a solution...
• coredns latency (compare to previous latency monitor which uses a threshold instead of a change alert)

kubelet

• rate updated
• error updated
• duration updated note: this metric is not calculating meaningful values because (I think) the collection window is not the same for both of them. From what I have gathered there is a flag --operation-duration-history-seconds in the kubelet config file that controls the sum metric; and there should be a metric aggregation interval in the datadog agent config file that sets a scraping interval. I am not sure how to access either of these files to check, though.

[jhouse-solvd]

This issue is nearly done but will stay open into the next sprint so that we can get it properly reviewed.

[jhouse-solvd]

Multiple team members have been out of the office recently, which has impacted our ability to complete this.

But now that @ph-One is back, he should be able to review and provide feedback, and then we'll get this over the line.

[npeterson54]

Not quite complete, rolling into next sprint

[npeterson54]

The Kublet duration is not making sense and needs to be investigated further. Potentially lets spin up another ticket.

[it-harrison]

Closing this ticket and breaking out the latency monitor into its own ticket.