search

jupyterhub / grafana-dashboards

Grafana Dashboards useful for k8s + JupyterHub
https://jupyterhub-grafana.readthedocs.io
BSD 3-Clause "New" or "Revised" License
54 stars 22 forks source link

Let JupyterHub Dashboard's "Users per node" report empty nodes as well #77

Open consideRatio opened 1 year ago

consideRatio commented 1 year ago

The JupyterHub Dashboard's "Users per node" panel should also report nodes with 0 users on them I think, otherwise one may think the k8s cluster doesn't have idle nodes when it does.

Example situation

The dashboard panel reports very few nodes.

image

But there are plenty of nodes (nodes with labels associated with being a user node etc), just that they don't have users on them.

image

Related

consideRatio commented 10 months ago

I think this is also a bug that relates to a bug of reporting 1 user when its in practice 0, just because there is one entry at one point in time reporting 1 followed by "no value".

Here is what I observed - note how the lines change but nothing ever goes below 1

staging-not-zero

EDIT: I think this is fixed by #90 as well.

consideRatio commented 5 months ago

I think the issue is with the query, which use pod info to acquire node info, instead of using node info to then count pods. Without a user pod scheduled on a node, it isn't recognized.

sum(
    # kube_pod_info.node identifies the pod node,
    # while kube_pod_labels.node is the metrics exporter's node
    kube_pod_info{node!=""}
    * on (namespace, pod) group_left() 
  group(
    kube_pod_labels{label_app="jupyterhub", label_component="singleuser-server", namespace=~"$hub"}
  ) by (label_component, pod, namespace)
) by (node)