search

carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 200 forks source link

Prometheus.<node_ip>.nip.io is unreachable #81

Closed nwber closed 4 years ago

nwber commented 4 years ago

Can't reach Prometheus/Grafana/Alertmanager. I'm running a k3s cluster of 4 Raspberry Pi 4s, each running Raspberry Pi OS Lite.

Screen Shot 2020-08-02 at 7 58 00 PM

My cluster is healthy and my nodes and master are all available.

Screen Shot 2020-08-02 at 8 10 04 PM

I've opened both ports 80 + 443 on both my master and node (via ufw).

The output of kubectl get ingress --all-namespaces and my vars.jsonnet file:

ingress+vars_monitoring_rpi

I've followed the steps for the k3s quickstart. There isn't much in the way of troubleshooting in the readme, so I'm a bit stuck.

carlosedp commented 4 years ago

Are you using K3s? Are the ingress pods (Traefik) running? Are you able to access any published application in the cluster, like a simple NGINX? Have you tried setting the ingress suffix with the same IP as the master IP?

nwber commented 4 years ago

Yes, I'm running k3s. Here are my pods. As you can see, I got the standard Kubernetes Dashboard working (ignore my-release-nginx, I accidentally tried using an x86 helm chart):

Screen Shot 2020-08-03 at 7 50 50 PM

I changed my ingress suffix to 10.0.0.9 and re-did the make & kubectl apply commands, still doesn't work:

Screen Shot 2020-08-03 at 8 04 12 PM

My Kubernetes Dashboard:

Screen Shot 2020-08-03 at 8 11 36 PM
carlosedp commented 4 years ago

I see that your master has the IP 10.0.0.9 and the ingresses point to 10.0.0.10.nip.io. Might be your problem. Other than that, you might need to investigate further on the infrastructure.

nwber commented 4 years ago

I figured it out: In the quickstart, I did the steps manually instead of make deploy. Running make deploy worked. Not sure why but it did 🤷‍♂️