[bug] Broken panels on k8s-views-nodes due to the nodename label

[vibhavachar]

Describe the bug

The dashbaords are really hepfull but ia m trying multi prometehus setup and i am unable to distinguish between different clusters. The jobs have the lablels, treid all relable configs but stil unable to get the node name metrics.

How to reproduce?

prometheus-community/kube-prometheus-stack --version 54.2.1

Expected behavior

able to see nodename info

Additional context

prometheus-node-exporter: namespaceOverride: "" podLabels:

Add the 'node-exporter' label to be used by serviceMonitor to match standard common usage in rules and grafana dashboards

##
jobLabel: ops-infra-tools-node-exporter

releaseLabel: true extraArgs:

• --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)

• --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ service: portName: http-metrics ipDualStack: enabled: false ipFamilies: ["IPv6", "IPv4"] ipFamilyPolicy: "PreferDualStack" prometheus: monitor: enabled: true

  jobLabel: jobLabel

  Scrape interval. If not set, the Prometheus default scrape interval is used.

  interval: ""

  SampleLimit defines per-scrape limit on number of scraped samples that will be accepted.

  sampleLimit: 0

  TargetLimit defines a limit on the number of scraped targets that will be accepted.

  targetLimit: 0

  Per-scrape limit on number of labels that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer.

  labelLimit: 0

  Per-scrape limit on length of labels name that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer.

  labelNameLengthLimit: 0

  Per-scrape limit on length of labels value that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer.

  labelValueLengthLimit: 0

  How long until a scrape request times out. If not set, the Prometheus default scape timeout is used.

  scrapeTimeout: ""

  proxyUrl: URL of a proxy that should be used for scraping.

  proxyUrl: ""

  MetricRelabelConfigs to apply to samples after scraping, but before ingestion.

  ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig

  metricRelabelings: []

  - sourceLabels: [name]

  separator: ;

  regex: ^node_mountstatsnfs(event|operations|transport)_.+

  replacement: $1

  action: drop

  RelabelConfigs to apply to samples before scraping

  ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig

  relabelings:

  • sourceLabels: [__meta_kubernetes_pod_node_name] separator: ; regex: ^(.*)$ targetLabel: nodename replacement: $1 action: replace

[NicoJDE]

same problem here.

when i remove this filter it works.

[logan-bobo]

@NicoJDE we have recently also ran into a similar problem. However ours was due to how Flatcar set the hostname for nodes. Resulting in the nodename label being the hostname with no domain information and $node is an FQDN

For me the fix was to change the variable definition to instance, this for us was the nodes FQDN. Then as the value in $node is the FQDN we can simply filter on $node showing the metrics for the currently selected node.

"label_values(node_uname_info{instance=~\"$node\"},instance)",

This is now working in our K8s setup across AWS, GCP and Azure using kube-prometheus. I will port this fix back upstream now.

[dotdc]

There's a nodename label known issue, did you checked it? Don't hesitate to make a PR if you find another isssue or limitation.

[logan-bobo]

Thanks I will take a look at this one as it might be a better fix to our current issue rather than vendoring and altering the dashboards.

[NicoJDE]

@dotdc thanks for the hint, this works for me

[dotdc]

Great, closing the issue!