geerlingguy / pi-cluster

Raspberry Pi Cluster automation
MIT License
476 stars 48 forks source link

port-forward for grafana hangs indefinitely: Grafana webpage not accessible #15

Open alehanderoo opened 5 months ago

alehanderoo commented 5 months ago

Hi @geerlingguy,

First of all, thank you for open-sourcing this! I’ve learned a lot about Ansible and server configuration over the last few days (and nights)! What a fantastic tool!

Describe the bug

When I then run kubectl port-forward service/cluster-monitoring-grafana :80 (as user and as root) the device does not finish the command and grafana is never accessible.

root@node1:/home/rock# kubectl port-forward service/cluster-monitoring-grafana :80
Forwarding from 127.0.0.1:46238 -> 3000
Forwarding from [::1]:46238 -> 3000

Opening http://192.168.2.52:46238/ does not return a page.

Troubleshooting

I'm running a self-built cluster. Control_plane on a rockpi4:

   Static hostname: rockpi4c
       Operating System: Ubuntu 20.04.6 LTS
            Kernel: Linux 4.4.154-112-rockchip-gfdb18c8bab17
      Architecture: arm64

Remaining 4 nodes: (Rpi4 and Rpi3)

Operating System: Debian GNU/Linux 12 (bookworm)
          Kernel: Linux 6.6.31+rpt-rpi-v8
    Architecture: arm64

Networking:

Main installation:

root@node1:/home/rock# kubectl get nodes
NAME          STATUS   ROLES                  AGE   VERSION
node5         Ready    <none>                 51m   v1.29.5+k3s1
node3         Ready    <none>                 51m   v1.29.5+k3s1
node4         Ready    <none>                 51m   v1.29.5+k3s1
node1   Ready    control-plane,master   52m   v1.29.5+k3s1
node2         Ready    <none>                 51m   v1.29.5+k3s1

root@node1:/home/rock# kubectl get pods
NAME                                                    READY   STATUS    RESTARTS   AGE
nfs-subdir-external-provisioner-7df9c8b467-256mg        1/1     Running   0          50m
cluster-monitoring-prometheus-node-exporter-mm9gw       1/1     Running   0          49m
cluster-monitoring-prometheus-node-exporter-2xm24       1/1     Running   0          49m
cluster-monitoring-prometheus-node-exporter-lf4hn       1/1     Running   0          49m
cluster-monitoring-prometheus-node-exporter-ggg5z       1/1     Running   0          49m
cluster-monitoring-prometheus-node-exporter-h4kps       1/1     Running   0          49m
cluster-monitoring-kube-state-metrics-df8db86bb-zq4lz   1/1     Running   0          49m
cluster-monitoring-kube-pr-operator-b44c59f5d-8qp84     1/1     Running   0          49m
cluster-monitoring-grafana-5b4dd85976-8cv2m             3/3     Running   0          49m
prometheus-cluster-monitoring-kube-pr-prometheus-0      2/2     Running   0          48m
alehanderoo commented 5 months ago

Got it working already! Posting it here for anyone having the same issue.

run kubectl edit svc cluster-monitoring-grafana -n default on control_plane node. This will show the following vi editor.

Change the type to NodePort and add the nodePort port.

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: cluster-monitoring
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-06-04T15:41:14Z"
  labels:
    app.kubernetes.io/instance: cluster-monitoring
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 10.4.1
    helm.sh/chart: grafana-7.3.11
  name: cluster-monitoring-grafana
  namespace: default
  resourceVersion: "14694"
  uid: 2b047274-31cf-413b-8dc2-14b8571a8330
spec:
  clusterIP: 10.43.27.129
  clusterIPs:
  - 10.43.27.129
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http-web
    nodePort: 30080 # Optional: specify a port, or leave it to let Kubernetes assign one
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app.kubernetes.io/instance: cluster-monitoring
    app.kubernetes.io/name: grafana
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

run kubectl get svc cluster-monitoring-grafana -n default to validate the settings.

NAME                         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
cluster-monitoring-grafana   NodePort   10.43.27.129   <none>        80:30080/TCP   4h27m
alehanderoo commented 5 months ago

Does not seem to work after a reboot.

BicycleJohny commented 2 months ago

It is probably because it is handeld by helm. I have the same issue and I am trying to convice helm to do it

BicycleJohny commented 2 months ago

UPDATE: Ok, it wasn't that hard after all. You can either extend this file on tasks[1].values with:

grafana:
    service:
      type: NodePort
      nodePort:30080

and uninstall with helm and reinstall it with ansible. Or you can just uninstall it with helm and put all the values into file like values.yml:

alertmanager:
  enabled: false
grafana:
  service:
    type: NodePort
    nodePort: 30080

And then install it again with helm:

helm install prometheus-stack prometheus-community/kube-prometheus-stack -f values.yaml --kubeconfig /etc/rancher/k3s/k3s.yaml

It then creates Grafana service with type NodePort accessible from specified port

NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
kubernetes                                  ClusterIP   10.43.0.1       <none>        443/TCP             40h
prometheus-operated                         ClusterIP   None            <none>        9090/TCP            13m
prometheus-stack-grafana                    NodePort    10.43.10.202    <none>        80:30080/TCP        13m
prometheus-stack-kube-prom-operator         ClusterIP   10.43.89.109    <none>        443/TCP             13m
prometheus-stack-kube-prom-prometheus       ClusterIP   10.43.61.6      <none>        9090/TCP,8080/TCP   13m
prometheus-stack-kube-state-metrics         ClusterIP   10.43.165.23    <none>        8080/TCP            13m
prometheus-stack-prometheus-node-exporter   ClusterIP   10.43.155.255   <none>        9100/TCP            13m
BicycleJohny commented 2 months ago

Btw @alehanderoo, kubectl port-forward is temporary thing and it should wait for termination from user - it creates temporary forward rule and waits until you are finish (ctrl+c). That is why it looks like it hangs