Observability for KaaS for the Operator

[fkr]

As a SCS Operator I need to have a clear understanding of the health of managed clusters on the KaaS layer.

Definition of Ready:

• [ ] User Story is small enough to be finished within one sprint
• [ ] User Story is clear and understood by the whole team
• [ ] Acceptance criteria are defined
• [ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

Preliminaries:

• [x] #299
• [x] #306
• [x] #300
• [x] #301

MVP0:

• [x] #303
  • [x] k8s cluster that hosts observer deployment is deployed
  • [x] s3 compatible bucket as a storage for long term metrics is configured
  • [x] thanos query-frontend is deployed and configured
  • [x] thanos query is deployed and configured
  • [x] thanos reciever is deployed and configured (simple deployment, non HA, without router)
  • [x] thanos ruler is deployed and configured
  • [x] thanos compactor is deployed and configured
  • [x] thanos bucket-web is deployed and configured
  • [x] thanos storegateway is deployed and configured
  • [x] prometheus server is deployed and configured
  • [x] prometheus alertmanager is deployed and configured
  • [x] prometheus black-box exporter is deployed and configured
  • [x] kaas-metric-importer is deployed and configured (service aims to differentiate between intentional deletion of KaaS instances and failures in the KaaS monitoring agent)
• [x] #305
  • [x] all prometheus alerts are working as expected
• [x] #304
  • [x] KaaS L0 dashboard counters are working correctly
  • [x] dedicated L0 dashboards are deployed for KaaS and for IaaS monitoring layers
• [x] #480
• [x] #482
• [x] #302
  • [x] KaaS service is mocked
  • [x] VM that will host a mock of KaaS service is deployed
  • [x] a script that deploys a multiple KinD clusters and register them in observer is created
• [x] #481
  • [x] IaaS service is mocked
  • [x] OSISM testbed is deployed
  • [x] implement an option to deploy thanos sidecar with some simple config in OSISM testbed
• [x] SovereignCloudStack/k8s-harbor#51
• [x] SovereignCloudStack/k8s-harbor#52

Checks:

• [x] Documentation has been updated
• [x] Release Notes have been updated

[matofeder]

Hi @fkr, @jschoone

I played a little bit with the OpenTelemetry project and SCS KaaS reference implementation. Please find my PoC (still WIP) here (Otel directory).

This PoC does:

• Patch SCS KaaS project to be able to create a k8s cluster that export traces of kube-apiserver, kubelet, and containerd to the deployed OpenTelemetry Collector
• Configure OpenTelemetry Collector to pull k8s cluster metrics
• Configure OpenTelemetry Collector to collect k8s pods logs
• Configure OpenTelemetry Collector to export mentioned traces, metrics, logs to the backends
• Deploy backends for all three signals (metrics, traces, logs): Prometheus, Jaeger, Loki
• Deploy Grafana with all three backends as datasources
• Adds docs around

we can discuss this in the next SIG Monitoring meeting 2023/7/7_12:05

[matofeder]

@fkr, @o-otte view the suggested issues for the upcoming phase of MVP (MVP1):

• [ ] https://github.com/SovereignCloudStack/issues/issues/449
• [ ] https://github.com/SovereignCloudStack/issues/issues/450
• [ ] https://github.com/SovereignCloudStack/issues/issues/451
• [ ] https://github.com/SovereignCloudStack/issues/issues/452
• [ ] https://github.com/SovereignCloudStack/issues/issues/495
• [ ] https://github.com/SovereignCloudStack/issues/issues/524
• [ ] https://github.com/SovereignCloudStack/k8s-observability/issues/32

[matofeder]

@fkr At today's SIG monitoring meeting, we discussed the DoD list proposal for MVP-0, https://input.scs.community/monitoring-mvp-0#. As @o-otte agreed with it, I would like to ask you whether I can edit the epic description according to the proposed DoD. Afterward, I will create issues from the DoD open items, and then the Gonicus team could easily contribute to MVP-0.

[fkr]

yes - i think, that makes a lot of sense!

[matofeder]

See the high-level architectural diagram, which illustrates the concept of MVP-0

EDIT(27/11/2023): An update that incorporated insights from SCS Sig Monitoring/SCS team OPS meeting

[matofeder]

All DoDs for MVP0 and beyond have been fulfilled as the last missing piece has been merged https://github.com/osism/testbed/pull/1855