ricsanfre
commented
2 years ago Procedure for obtaining raw metrics exposed by K3S.
The procedure described here https://github.com/SUSE/doc-caasp/issues/166#issuecomment-476191064 can be used to manually query https metrics endpoints. Most recent versions of Kubernetes are moving all metrics endpoint to use https.
For example: TCP ports numbers exposed by kube-scheduler and kube-controller-manager have changed from kubernetes release 1.22 (from 10251/10252 to 10257/10259) and now require https authenticated connection. Kubernetes authorized service account is needed. Only kube-proxy endpoint remains open using HTTP, the rest of the ports are now using HTTPS
The procedure specified above creates a service account with not enough privileges to query directly kubelet metrics endpoints. The following service account, role and rolebinding resources need to be created:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: monitoring
namespace: kube-system
secrets:
- name: monitoring-secret-token
---
apiVersion: v1
kind: Secret
metadata:
name: monitoring-secret-token
namespace: kube-system
annotations:
kubernetes.io/service-account.name: monitoring
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: monitoring-clusterrole
namespace: kube-system
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- pods
verbs: ["get", "list"]
- nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: monitoring-clusterrole-binding
namespace: kube-system
roleRef:
kind: ClusterRole
name: monitoring-clusterrole
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: monitoring
namespace: kube-systemFollowing script can be used to automatically extract metrics from kubelet, kube-proxy and apiserver endpoints and compare the results:
#!/bin/bash
# Get token
TOKEN=$(kubectl -n kube-system get secrets monitoring-secret-token -ojsonpath='{.data.token}' | base64 -d)
APISERVER=$(kubectl config view | grep server | cut -f 2- -d ":" | tr -d " ")
# Get apiserver
curl -ks $APISERVER/metrics --header "Authorization: Bearer $TOKEN" | grep -v "# " > apiserver.txt
# Get list of nodes of k3s cluster from api server and iterate over it
for i in `kubectl get nodes -o json | jq -r '.items[].status.addresses[0].address'`; do
echo "Getting metrics from node $i"
curl -ks https://$i:10250/metrics --header "Authorization: Bearer $TOKEN" | grep -v "# " > kubelet_$i.txt
curl -ks https://$i:10250/metrics/cadvisor --header "Authorization: Bearer $TOKEN" | grep -v "# " > kubelet_cadvisor_$i.txt
curl -ks http://$i:10249/metrics | grep -v "# " > kubeproxy_$i.txt
done
# Get kube-controller and kube-scheduler
for i in `kubectl get nodes -o json | jq -r '.items[] | select(.metadata.labels."node-role.kubernetes.io/master" != null) | .status.addresses[0].address'`; do
echo "Getting metrics from master node $i"
curl -ks https://$i:10259/metrics --header "Authorization: Bearer $TOKEN" | grep -v "# " > kube-scheduler_$i.txt
curl -ks https://$i:10257/metrics --header "Authorization: Bearer $TOKEN" | grep -v "# " > kube-controller_$i.txt
doneAnalyzing the results
Executing the previous script, the following files contains the metrics extracted from each of the exposed ports in each of the nodes of the cluster:
apiserver.txt kube-controller_node1.txt kubelet_cadvisor_node1.txt kubelet_cadvisor_node2.txt kubelet_cadvisor_node3.txt kubelet_cadvisor_node4.txt kubelet_node1.txt kubelet_node2.txt kubelet_node3.txt kubelet_node4.txt kubeproxy_node1.txt kubeproxy_node2.txt kubeproxy_node3.txt kubeproxy_node4.txt kube-scheduler_node1.txt
-
Checking metrics extracted from node1 (master) endpoints, all ports are exposing the same number of metrics:
~$ wc -l kubelet_node1.txt 40666 kubelet_node1.txt ~$ wc -l kubeproxy_node1.txt 40666 kubeproxy_node1.txt ~$ wc -l kube-controller_node1.txt 40666 kube-controller_node1.txt ~$ wc -l kube-scheduler_node1.txt 40666 kube-scheduler_node1.txt ~$ wc -l apiserver.txt 40666 apiserver.txtThe metrics in the files are the same, when applying
diffcommand the only differences showed are the values in some of the metrics (counters/seconds). This is due to that the different ports are polled in different times so, the counter of seconds type metric is showing different values -
Checking metrics extracted from node2 (worker) endpoints, all ports are exposing the same number of metrics:
~$ wc -l kubelet_node2.txt 1723 kubelet_node2.txt ~$ wc -l kubeproxy_node2.txt 1723 kubeproxy_node2.txtand again the differences are only in the values of counters(seconds) type metrics.
Conclusion
To get all k3s metrics it is enough with collecting metrics from kubelet endpoints (/metrics, /metrics/cadvisor and /metrics/probe) in all nodes
sherif-fanous
mrclrchtr
Bug Description
Kuberentes Documentation - System Metrics details which Kubernetes components expose metrics in Prometheus format:
These components are:
/metricsendpoint at TCP 10257)/metricsendpoint at TCP 10249)/metricsat Kubernetes API port)/metricsendpoint at TCP 10259)/metrics,/metrics/cadvisor,/metrics/resourceand/metrics/probesendpoints at TCP 10250)K3S distribution has a special behavior since in each node only one process is deployed (
k3s-serverrunning on master nodes ork3s-agentrunning on worker nodes) with all k8s components sharing the same memory.K3s is emitting the same metrics, from all k8s components deployed in the node, at all '/metrics' endpoints available (api-server, kubelet (TCP 10250), kube-proxy (TCP 10249), kube-scheduler (TCP 10251), kube-controller-manager (TCP 10257). Thus, collecting from all port produces metrics duplicates.
kubelet additional metrics (endpoints
/metrics/cadvisor,/metrics/resourceand/metrics/probes) are only available at TCP 10250.Enabling the scraping of all different metrics TCP ports (kubernetes components) causes the ingestion of duplicated metrics. Duplicated metrics in Prometheus need to be removed in order to reduce memory and CPU consumption.
Context Information
As stated in issue #22, there was a known issue in K3S: https://github.com/k3s-io/k3s/issues/2262, where duplicated metrics are emitted by the three components (kube-proxy, kube-scheduler and kube-controller-manager). The proposed solution by Rancher Monitoring(https://github.com/k3s-io/k3s/issues/2262), was to avoid the scrape of duplicated metrics and activate only the service monitoring of one of the components. (i.e. kube-proxy). That solution was implemented (see https://github.com/ricsanfre/pi-cluster/issues/22#issuecomment-986224709) and it solved the main issue (out-of-memory).
Endpoints currently being scrapped by Prometheus are
Duplicated metrics
After deeper analysis on the metrics scrapped by Prometheus, it is clear that K3S is emitting duplicated metrics in all endpoints.
Example 1: API-server metrics emitted by kube-proxy, kubelet and api-server endpoints running on master server
Example 2: kubelet metrics emitted by kube-proxy, kubelet and api-server
Example3: kubepoxy metrics:
kubeproxy_sync_proxy_rules_duration_seconds_bucket{le="0.001"}