search

carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
739 stars 200 forks source link

Trouble with node exporters in rasperrypies with public ip #94

Closed exArax closed 4 years ago

exArax commented 4 years ago

I am setting up a cluster with public ip. The k3s master is a cloud vm and I have two rpis as workers all of them they have public ip, so they are not in the same LAN. The node exporters in the rpis have a down status and they show a context deadline exceeded error. I cannot find what's going wrong. The weird fact is that the cadvisors of rpis have up statuses, and all the required ports are open.

carlosedp commented 4 years ago

Prometheus scrapes node_exporter from all nodes using internal IPs from the node_exporter pods/services.

When you say "down status" you mean in Prometheus Targets?

exArax commented 4 years ago

@carlosedp yes exactly

carlosedp commented 4 years ago

First of all, I'd advise against having your nodes exposed to the internet with valid IPs, but it shouldn't affect the usage.

We'd need more information:

  1. Is there any firewall between these LANs? Do they allow the traffic between Prometheus and the node_exporter port (9100)?
  2. Do you have any log from node_exporter? Is it running correctly (both containers, node_exporter and kube-rbac-proxy)?
  3. Can you try to curl -k one of the IPs in port 9100? It will fail since there is no authentication but that's fine.
exArax commented 4 years ago

When I am doing curl from the host I am getting unauthorized message, if I perform it from another device I am getting timed out message. I will try to find answers to the other two questions and I will post them here, thank you @carlosedp

carlosedp commented 4 years ago

Apparently there is something blocking the access between the hosts on the needed port since curl didn't work from outside.

exArax commented 4 years ago

In the ufw of the devices I have allowed the required ports. These are the logs for node-exporter and kube-rbac-proxy: image Untitled

exArax commented 4 years ago

I changed the port of the node-exporters and now it works !!! @carlosedp thank you for your time :)