5. Improve Monitoring Stack

[abhi4578]

• Add and improve existing dashboards according to give more insight on the happening in the K8s cluster, each of the components
• Add more alerts for detection of application/system failure

[abhi4578]

K8s Cluster and addons Monitoring

• Add k8s pvc metrics dashboard, - to monitor pvc usage and alert before reaches the full disk capactity

  • this looks to be good, dashboard 13646

• Etcd monitoring

• Autoscaling monitoring

  • Cluster autsocaler dashboard 3831
  • Horizontal pod autoscaler dashboard 10257

• velero backups monitoring

• sealed secrets

  • refer

• nginx ingress

Other resources:

• Awesome monitoring mixins : https://monitoring.mixins.dev/
• Intro to Monitoring Kubernetes with grafana cloud: https://grafana.com/go/webinar/intro-to-monitoring-kubernetes/?pg=hp&plcmt=upcoming-events-2
• Bitnami grafana dashnoards for cluster - need to tune
• Devopsprodigy - need to try out

[abhi4578]

Backend Components

1. Elasticsearch
2. Redis
3. Postgres
4. Rabbitmq
5. Immudb
   • Enable metrics and add scrape annotation.
   • Dashobard 12026

[abhi4578]

API Servers

1. Blackbox testing - update to work with k8s and new auth implementation
2. improve vertx and jvm dashboards
3. improve log monitoring of api servers