Requirements of the operators regarding KaaS Monitoring

[o-otte]

We need to know, what exactly an Operator needs to see in order to provide a stable, performant managed Kubernetes to their customers:

• how is a stable managed k8s defined
• how is a perfomaned managed k8s defined
• what is a managed kubernetes
• where does it start and stop => Team KaaS needs to deliver this
• Feedback Loop Operators
  • Question to Operators of existing KaaS Offerings:
  • How deep are you looking?
    • What kind of data are you allowed to access and what is helpful to access in order to assure that your service is operating fully functional?
  • Do we want to aggregate logs? Are logs needed (kubelet logs)
    • Does this play into the audit logging as well?
    • What kind of actions are relevant for audit logging
    • Is logging a separate epic? (no, since the epic is called 'Observability')
    • would it help us to "solve" logging on IaaS first?

How do we want to gather the input from our operators?

• usually we do this in a team call, but maybe a different way this time?
  • questionaire?
  • Wishlist
• this does not solve, finding out WHO to ask
  • GSK (gridscale managed kubernetes)
  • PSKE (ask PO to interface with PSKE team)
  • Wavestack
  • OTC
  • 23 Tech
  • dNation
  • syself
  • cleura

The outcome should be an ADR on the requirements as a foundation of the following work on the corresponding epic.

Definition of Ready:

• [ ] User Story is small enough to be finished within one sprint
• [ ] User Story is clear and understood by the whole team
• [ ] Acceptance criteria are defined
• [ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

• [ ] All acceptance criteria are met
• [ ] ADR is written
• [ ] Changes have been reviewed
• [ ] CI tests have run successfully
• [ ] Documentation has been updated
• [ ] Release Notes have been updated

[o-otte]

Preliminary Survey Results:

1. What is your understanding of a managed Kubernetes Offering:

   • Hassle-Free Installation and Maintainance (customer viewpoint); Providing Controlplane and worker nodes and responsibility for correct function but agnostic to workload
   • Day0, 1 and 2 (~planning, provisioning, operations) full lifecyle management or let customer manages some parts of that, depending on customer contract

2. What Type and Depth of observability is needed

   • CPU, RAM, HDD and Network usage, Health and Function of Cluster Nodes, Controlplane and if desired Customer Workload

3. Do you have an observabiltiy infrastructure, if yes, how it is built

   • Grafana/Thanos/Prometheus/Loki/Promtail/Alertmanger Stack, i.e. https://raw.githubusercontent.com/dNationCloud/kubernetes-monitoring-stack/main/thanos-deployment-architecture.svg

4. Data Must haves

   • CPU, RAM, Disk, Network
   • HTTP Connectivity Metrics
   • Control Plane and Pod metrics (States, Ready, etc.)
   • Workload specific metrics
   • node stats
   • k8s resources (exporters, kubestate metrics, cadvisor, parts of the kubelet)
   • ingress controller exporter (http error rate, cert metrics like expiration date)
   • k8s certs metrics
   • metrics of underlying node
   • Logs of control plane, kubelet and containerd

5. Must Not haves

   • Secrets, otherwise as much as possible for anomaly detection over long time data

6. Must have Alerts

   • Dependent on SLAs and SLA Types, highly individual
   • use of https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/alerts and https://github.com/dNationCloud/kubernetes-monitoring/tree/main/jsonnet/rules

7. Must NOT Alert on

   • Should not wake people, nothing that does not lead to Action items

8. Observability from Within Or Outside KaaS. How does the architecture look like?

   • Monitoring Infra on CSP Side
   • Data from Customer Clusters and Mon Infra on CSP and KaaS, get both data. KaaS Monitoring can also be used by customer

9. Special Constraints

   • HA Setup in different Clusters on Different Sites