Deploy k8 alerting into production

[jeremyw24]

Goals:

1. Metrics
2. Logs
3. Notifications

Stage 1:

What's the role of NewRelic today?

• Evaluate whether to use it as a standard log + metrics aggregator for all clusters? Replace Grafana.
• Or would that too expensive
  • network costs
  • subscrition costs
  • @viv-4 to research and estimate this cost and effectiveness, features, UI NR + AKS egress 3 SaaS
  • Create a brief enternal gdoc with this evaluation by mid Jan.

Later, Stage 2:

If not NewRelic, Which platforms to use?

Metrics

Prometheus + Grafana

Logs

Logspout > Grafana?: But is Grafana good for log aggregtion and searching

Notifications

[viv-4]

Diagram of planned SaaS monitoring based on how we believe the pieces will work together: