Closed geerlingguy closed 4 years ago
Should this issue be opened against the arm_exporter build repo? https://github.com/carlosedp/docker-arm_exporter
Or... against the upstream rpi_exporter repo? https://github.com/lukasmalkmus/rpi_exporter
It seems like it would be the responsibility of the arm_exporter since it's K8s-specific and we might have to tweak rpi_exporter to be able to get the node IP instead of the pod IP.
It looks like there's a related issue here: https://github.com/lukasmalkmus/rpi_exporter/issues/2
Looking thru this.
Can you check if https://github.com/carlosedp/cluster-monitoring/pull/45 fixes this?
@carlosedp - Testing now. Thanks!
Things seemed to be good, but after a few minutes I get a CrashLoopBackoff. I've been doing a ton of messing around, though, so it could be something on my end. I'm going to test out your change to #42 to make it work better out of the box, then later I'm going to rebuild the cluster and test this again.
Could you check the logs for the error that caused the crashloopback? Might not be related to this... but who knows :)
The logs looked like the pods started up fine, that's why I think it might be some other misconfiguration I had. I'm re-building the configs now with this PR applied, and I'll see if it works this time around.
@carlosedp - Strange, it's still doing the CrashLoopBackOff
again with this change after a fresh rebuild of the cluster. Here are the logs from one of the crashing pods:
# kubectl logs -n monitoring arm-exporter-4ccdw -c arm-exporter
time="2020-05-25T18:15:50Z" level=info msg="Starting rpi_exporter(version=, branch=, revision=)" source="rpi_exporter.go:82"
time="2020-05-25T18:15:50Z" level=info msg="Build context(go=go1.14.2, user=, date=)" source="rpi_exporter.go:83"
time="2020-05-25T18:15:50Z" level=info msg="Listening on127.0.0.1:9243" source="rpi_exporter.go:115"
Ah... it looks like that pod is fine, the kube-rbac-proxy
pod is now failing:
# kubectl logs -n monitoring arm-exporter-4ccdw -c kube-rbac-proxy
I0525 18:20:07.060798 1 main.go:186] Valid token audiences:
I0525 18:20:07.061127 1 main.go:232] Generating self signed cert as no cert is provided
I0525 18:20:21.490236 1 main.go:281] Starting TCP socket on 10.0.100.163:9243
F0525 18:20:21.490836 1 main.go:284] failed to listen on secure address: listen tcp 10.0.100.163:9243: bind: cannot assign requested address
That container can't set --secure-listen-address
to the host IP, only the pod IP.
Ah yes, let me try another thing...
Latest push to the PR #45 might do it.
Testing.
Rbac proxy seems to be working now, here are its logs:
# kubectl logs -n monitoring arm-exporter-x65cm -c kube-rbac-proxy
I0525 19:18:51.194470 1 main.go:186] Valid token audiences:
I0525 19:18:51.196550 1 main.go:232] Generating self signed cert as no cert is provided
I0525 19:19:06.708033 1 main.go:281] Starting TCP socket on 10.42.0.25:9243
I0525 19:19:06.709232 1 main.go:288] Listening securely on 10.42.0.25:9243
2020/05/25 19:19:11 http: proxy error: dial tcp 127.0.0.1:9243: connect: connection refused
But the other container is now failing, if I do a kubectl describe pod
on it, I get:
# kubectl describe pod arm-exporter-x65cm -n monitoring
...
Normal Created 23s (x4 over 72s) kubelet, turing-master Created container arm-exporter
Warning Failed 22s (x4 over 71s) kubelet, turing-master Error: failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/bin/sh -c\": stat /bin/sh -c: no such file or directory": unknown
It looks like that container doesn't have sh
on it.
Tried another strategy... just pushed. Sorry about this trial and error... it might take some time til I can get an arm board to test this myself :)
Haha, no problem. I only work on the ARM-based stuff in bursts, and right now I have two clusters running for some tests, in addition to a couple other Pis, and so I'm happy to test! Working on testing this config now.
They all start now, but it's still reporting the pod IPs:
I confirmed that the deployed manifest was correct:
volumeMounts:
- mountPath: /etc/nodename
name: hostname
readOnly: true
And I even confirmed the mount is reporting the correct hostname:
# kubectl exec -n monitoring arm-exporter-4wstw -c arm-exporter cat /etc/nodename
worker-04
Could there be a setting in the Proemetheus config that's using the wrong value? Nope, see next comment. It seems to be something with rpi_exporter not picking up the /etc/nodename
?
Grafana's getting the following data:
__name__:"rpi_cpu_temperature_celsius"
endpoint:"https"
instance:"10.42.0.26:9243"
job:"arm-exporter"
namespace:"monitoring"
pod:"arm-exporter-rzg8k"
service:"arm-exporter"
And the label is {{instance}}
, so it's not something that could be tweaked on Grafana's side.
And in Prometheus, that's all it's getting too:
rpi_cpu_temperature_celsius{endpoint="https",instance="10.42.0.26:9243",job="arm-exporter",namespace="monitoring",pod="arm-exporter-rzg8k",service="arm-exporter"}
Exporter build info:
rpi_exporter_build_info{endpoint="https",goversion="go1.14.2",instance="10.42.0.26:9243",job="arm-exporter",namespace="monitoring",pod="arm-exporter-rzg8k",service="arm-exporter"}
The exporter should be adding a tag with node hostname... to match the other collectors. Give me some time to check on this
Got it now. Added a relabeling rule so the node name replaces the IP. Solution is easier than thought :)
Testing it out!
It works!!! Thanks!
Awesome! I've set a K3s server on an ARM board I had lying here to avoid bothering you... heheeh Had a spare SDcard with a ready OS.
Nice! Well, at this point all the minor nitpicks are working perfectly, I'm planning on featuring this as part of the next Turing Pi cluster video on my YouTube, so thanks for making it a perfect demo instead of a 'mostly-perfect' demo :)
I noticed the node-exporter seems to be giving back node IPs instead of Kubernetes Pod IPs, so when the dashboard is displayed in Grafana, I see my node DNS names, like worker-01, worker-02, etc.
The CPU temperature monitor data, though, uses Pod IPs instead of node IPs, so the data is a little harder to discern, since I have to manually map Pod IPs to the nodes those Pods are running on: