Investigate the use of kube-rbac-proxy on K3s

[carlosedp]

Dive deeper on kube-rbac-proxy use of the API on K3s to return endpoints into https and authenticated endpoints.

Ref. https://github.com/carlosedp/cluster-monitoring/issues/13

[hlugt]

As replied in https://github.com/rancher/k3s/issues/425 I tried the new manifests (I only changed the external addresses to my own). I now do have node-exporter connections and metrics, but the k3s kubelet and cadvisor still give connection refused ("Get http://#.#.#.#:10255/metrics: dial tcp #.#.#.#:10255: connect: connection refused" and "Get http://#.#.#.#:10255/metrics/cadvisor: dial tcp #.#.#.#:10255: connect: connection refused").

I just updated to the latest release of K3S: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.10.2 INSTALL_K3S_EXEC="--no-deploy traefik --no-deploy=servicelb" sh -s - --kubelet-arg="authentication-token-webhook=true" --kubelet-arg="authorization-mode=Webhook" --kubelet-arg="address=0.0.0.0"

Maybe I should by now get rid of the extra kubelet arguments?

(edit1: apologies for the late response, but I did not find enough time to investigate until now)

(edit2: when changing endpoint to https-metrics I get: "server returned HTTP status 400 Bad Request") (edit3: should have also changed the scheme to https ofcourse... But now I get: "Get https://#.#.#.#:10250/metrics/cadvisor: x509: certificate is valid for 127.0.0.1, not #.#.#.#")

(edit4: got targets now working. Need to collect some statistics first and then have a look at the dashoboards to see everything is indeed correct. Changed the prometheus-serviceMonitorKubelet.yaml to also contain the certificate and some to what I found mentioned at https://morioh.com/p/655b8bcb6747: (do not know ho to prevent formatting?) apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: kubelet name: kubelet namespace: monitoring spec: endpoints:

• bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: https-metrics scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
• bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token honorLabels: true interval: 30s path: /metrics/cadvisor port: https-metrics scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true jobLabel: k8s-app namespaceSelector: matchNames:
  • kube-system selector: matchLabels: k8s-app: kubelet

[carlosedp]

Latest version supports the use of kube-rbac-proxy on kube-state-metrics and node_exporter.