search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.56k stars 772 forks source link

Prometheus metrics of dqlite #4647

Open developer1622 opened 3 months ago

developer1622 commented 3 months ago

Summary

Note: I am unsure if this has been asked; I want to know if microk8s exposes Prometheus metrics of dqlite. As source dqlite, Does it expose Prometheus metrics?

When we install microk8s, by default, it is HA enabled, and we would have dqlite as storage; however, if at all we want to use etcd, we need to disable ha-cluster, then we would get etcd on port 12379, we can get the metrics of etcd by supplying TLS certificates.

So, in the same way, can we get the dqlite prometheus metrics?

Why is this important?

Every other K8s distro exposes metrics to understand better the things related to cluster storage.

Are you interested in contributing to this feature?

No

louiseschmidtgen commented 2 months ago

Hello @developer1622,

there are metrics collected on the k8s-dqlite layer (former kine layer that translates k8s requests to dqlite) or you can dive deeper into dqlite logs.

K8s-dqlite arguments

You can configure k8s-dqlite to collect metrics by passing arguments, their description and defaults are here: https://github.com/canonical/k8s-dqlite/blob/master/docs/configuration.md.

The metrics endpoint allows you to view the metrics of the k8s-dqlite layer with Prometheus. In MicroK8s 1.31 you will need to enable the metrics endpoint first before scraping the metrics.

Since MicroK8s 1.31, otel is our newest addition that allows you to gather insights via traces on queries to dqlite using a tool like Jaeger.

You can configure these flags by running sudo vim /var/snap/microk8s/current/args/k8s-dqlite and restarting k8s-dqlite: sudo snap restart microk8s.daemon-k8s-dqlite.

Dqlite logs

Currently, we don't have an endpoint to gather metrics on dqlite itself. If you need further insights on the underlying Raft protocol (node consensus) I would recommend you have a look at the logs as follows:

Optional: Add debug log to dqlite:

sudo vim /var/snap/microk8s/current/args/k8s-dqlite-env

uncomment LIBDQLITE_TRACE=1 and LIBRAFT_TRACE=1.

restart k8s-dqlite with the flags: sudo snap restart microk8s.daemon-k8s-dqlite.

Check dqlite logs: sudo journalctl -u snap.microk8s.daemon-k8s-dqlite -f.

These options will help you to gather further insights into dqlite. However, we encourage you to provide further feedback to help us improve our visibility into dqlite!