search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
10.64k stars 2.3k forks source link

The node_memory_Cached_bytes metric collected by Node Exporter differs by an order of magnitude compared to the value observed using the free command. #3033

Closed woodoow closed 1 month ago

woodoow commented 1 month ago

Host operating system: output of uname -a

Linux xxx 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter command line flags

node_exporter log output

Jun 01 23:21:15 systemd[1]: Stopping node_export... Jun 01 23:21:15 systemd[1]: Stopped node_export. Jun 01 23:21:38 systemd[1]: Started node_export. Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.098Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.1, branch=HEAD, revision=400c397993 Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.098Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.3, platform=linux/amd64, user=root@7afbf Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.098Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unpriv Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.099Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" fla Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:111 level=info msg="Enabled collectors" Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=arp Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=bcache Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=bonding Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=btrfs Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=conntrack Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=cpu Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=cpufreq Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.100Z caller=node_exporter.go:118 level=info collector=diskstats Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=dmi Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=edac Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=entropy Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=fibrechannel Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=filefd Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=filesystem Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=hwmon Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=infiniband Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=ipvs Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=loadavg Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=mdadm Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=meminfo Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=netclass Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=netdev Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=netstat Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=nfs Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=nfsd Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=nvme Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=os Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=powersupplyclass Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=pressure Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=rapl Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=schedstat Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=selinux Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=sockstat Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=softnet Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=stat Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=tapestats Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=textfile Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=thermal_zone Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=time Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=timex Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=udp_queues Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=uname Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=vmstat Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=watchdog Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=xfs Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=node_exporter.go:118 level=info collector=zfs Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100 Jun 01 23:21:38 node_exporter[520]: ts=2024-06-01T15:21:38.101Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100

Are you running node_exporter in Docker?

What did you do that produced an error?

[root@www node_exporter]# free -b total used free shared buff/cache available Mem: 8201244672 2074763264 382656512 383348736 5743824896 5429821440

What did you expect to see?

The actual value of buffer and the collected value differ by an order of magnitude.

What did you see instead?

HELP node_memory_Cached_bytes Memory information field Cached_bytes.

TYPE node_memory_Cached_bytes gauge

node_memory_Cached_bytes 5.14207744e+08

woodoow commented 1 month ago

[root@ node_exporter]# ./node_exporter --version node_exporter, version 1.8.1 (branch: HEAD, revision: 400c3979931613db930ea035f39ce7b377cdbb5b) build user: root@7afbff271a3f build date: 20240521-18:36:22 go version: go1.22.3 platform: linux/amd64 tags: unknown

SuperQ commented 1 month ago

What do you get for cat /proc/meminfo at the same time as when you capture the metrics?

woodoow commented 1 month ago

What do you get for cat /proc/meminfo at the same time as when you capture the metrics?

/proc/meminfo [root@www ~]# cat /proc/meminfo MemTotal: 8009028 kB MemFree: 326216 kB MemAvailable: 5335908 kB Buffers: 0 kB Cached: 566940 kB SwapCached: 8292 kB Active: 1297436 kB Inactive: 1048700 kB Active(anon): 1178748 kB Inactive(anon): 967744 kB Active(file): 118688 kB

The value for Cached 566940 kB, is incorrect.What can I do about it?

SuperQ commented 1 month ago

That is what the kernel reports, so if you can prove it's wrong, it's a kernel bug. Either way, the node_exporter only reports what the kernel tells it to. The issue is not in the node_exporter.