prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
10.64k stars 2.3k forks source link

Exporter may freeze if network file system is unavailable #3058

Open vitoyucepi opened 2 weeks ago

vitoyucepi commented 2 weeks ago

Host operating system: output of uname -a

Linux vm-ubuntu-22 5.15.0-1060-kvm #65-Ubuntu SMP Tue May 21 09:31:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.1 (branch: HEAD, revision: 400c3979931613db930ea035f39ce7b377cdbb5b)
  build user:       root@7afbff271a3f
  build date:       20240521-18:36:22
  go version:       go1.22.3
  platform:         linux/amd64
  tags:             unknown

node_exporter command line flags

--collector.filesystem.mount-timeout=1s --log.level=debug

node_exporter log output

Log ``` ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.1, branch=HEAD, revision=400c3979931613db930ea035f39ce7b377cdbb5b)" ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.3, platform=linux/amd64, user=root@7afbff271a3f, date=20240521-18:36:22, tags=unknown)" ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required." ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:199 level=debug msg="Go MAXPROCS" procs=1 ts=2024-06-21T20:33:15.059Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) ts=2024-06-21T20:33:15.059Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ ts=2024-06-21T20:33:15.059Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:111 level=info msg="Enabled collectors" ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=arp ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=bcache ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=bonding ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=btrfs ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=conntrack ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=cpu ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=cpufreq ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=diskstats ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=dmi ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=edac ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=entropy ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=fibrechannel ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=filefd ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=filesystem ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=hwmon ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=infiniband ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=ipvs ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=loadavg ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=mdadm ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=meminfo ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=netclass ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=netdev ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=netstat ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=nfs ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=nfsd ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=nvme ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=os ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=powersupplyclass ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=pressure ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=rapl ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=schedstat ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=selinux ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=sockstat ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=softnet ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=stat ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=tapestats ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=textfile ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=thermal_zone ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=time ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=timex ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=udp_queues ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=uname ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=vmstat ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=watchdog ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=xfs ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=zfs ts=2024-06-21T20:33:15.060Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100 ts=2024-06-21T20:33:15.060Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100 ts=2024-06-21T20:33:19.760Z caller=node_exporter.go:79 level=debug msg="collect query:" filters="unsupported value type" ts=2024-06-21T20:33:19.761Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=0.77 ts=2024-06-21T20:33:19.761Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=0.37 ts=2024-06-21T20:33:19.761Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=0.14 ts=2024-06-21T20:33:19.761Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=8.0849e-05 ts=2024-06-21T20:33:19.761Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded" ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=2.6859e-05 err="collector returned no data" ts=2024-06-21T20:33:19.761Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=8.222e-05 ts=2024-06-21T20:33:19.761Z caller=nvme_linux.go:56 level=debug collector=nvme msg="nvme statistics not found, skipping" ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=nvme duration_seconds=3.0973e-05 err="collector returned no data" ts=2024-06-21T20:33:19.761Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system" ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=3.3635e-05 err="collector returned no data" ts=2024-06-21T20:33:19.761Z caller=fibrechannel_linux.go:101 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping" ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=1.9921e-05 err="collector returned no data" ts=2024-06-21T20:33:19.761Z caller=hwmon_linux.go:440 level=debug collector=hwmon msg="hwmon collector metrics are not available for this system" ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=hwmon duration_seconds=1.4036e-05 err="collector returned no data" ts=2024-06-21T20:33:19.761Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu ts=2024-06-21T20:33:19.761Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io ts=2024-06-21T20:33:19.761Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory ts=2024-06-21T20:33:19.761Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=0.000140935 ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.000456171 ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=0.000107346 ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=9.826e-05 ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=9.995e-06 ts=2024-06-21T20:33:19.762Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping" ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=2.6714e-05 err="collector returned no data" ts=2024-06-21T20:33:19.762Z caller=rapl_linux.go:77 level=debug collector=rapl msg="Platform doesn't have powercap files present" err="unable to read class/powercap: open /sys/class/powercap: no such file or directory" ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=rapl duration_seconds=3.9654e-05 err="collector returned no data" ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.000176827 ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=8.7765e-05 ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=1.7034e-05 ts=2024-06-21T20:33:19.762Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory" ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=3.8249e-05 err="collector returned no data" ts=2024-06-21T20:33:19.762Z caller=watchdog.go:106 level=debug collector=watchdog msg="Could not read watchdog stats" err="failed to list watchdog devices at \"/sys/class/watchdog\": open /sys/class/watchdog: no such file or directory" ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=watchdog duration_seconds=1.9554e-05 err="collector returned no data" ts=2024-06-21T20:33:19.763Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.7190019997630088e+09 ts=2024-06-21T20:33:19.763Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=0 time_zone=UTC ts=2024-06-21T20:33:19.763Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [kvm-clock tsc] kvm-clock}]" ts=2024-06-21T20:33:19.763Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=0.000139335 ts=2024-06-21T20:33:19.763Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=3.3409e-05 ts=2024-06-21T20:33:19.763Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=1.7618e-05 ts=2024-06-21T20:33:19.763Z caller=schedstat_linux.go:78 level=debug collector=schedstat msg="schedstat file does not exist" ts=2024-06-21T20:33:19.763Z caller=collector.go:167 level=debug msg="collector returned no data" name=schedstat duration_seconds=2.2134e-05 err="collector returned no data" ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.001059162 ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=1.0888e-05 ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=7.236e-06 ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=3.8592e-05 ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=0.00013394 ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=2.6597e-05 ts=2024-06-21T20:33:19.764Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping" ts=2024-06-21T20:33:19.764Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=1.7543e-05 err="collector returned no data" ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=4.906e-05 ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000158008 ts=2024-06-21T20:33:19.765Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs ts=2024-06-21T20:33:19.765Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available" ts=2024-06-21T20:33:19.765Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=3.2581e-05 err="collector returned no data" ts=2024-06-21T20:33:19.765Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.000605296 ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0 ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0 ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1 ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5 ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5 ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.000673482 ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=8.0562e-05 ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=7.1122e-05 ts=2024-06-21T20:33:19.766Z caller=mdadm_linux.go:115 level=debug collector=mdadm msg="Not collecting mdstat, file does not exist" file=/proc ts=2024-06-21T20:33:19.766Z caller=collector.go:167 level=debug msg="collector returned no data" name=mdadm duration_seconds=3.6268e-05 err="collector returned no data" ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=7.5467e-05 ts=2024-06-21T20:33:19.766Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory" ts=2024-06-21T20:33:19.766Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=2.4558e-05 err="collector returned no data" ts=2024-06-21T20:33:19.766Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type" ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.000131379 ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=6.625e-06 ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=2.6535e-05 ts=2024-06-21T20:33:19.766Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net ts=2024-06-21T20:33:19.766Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=3.2812e-05 err="collector returned no data" ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=9.32e-06 ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service ts=2024-06-21T20:33:19.768Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=0.004034153 ts=2024-06-21T20:33:21.538Z caller=node_exporter.go:79 level=debug msg="collect query:" filters="unsupported value type" ts=2024-06-21T20:33:21.540Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=0.77 ts=2024-06-21T20:33:21.540Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=0.37 ts=2024-06-21T20:33:21.540Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=0.14 ts=2024-06-21T20:33:21.540Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=0.000301563 ts=2024-06-21T20:33:21.540Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system" ts=2024-06-21T20:33:21.541Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=0.000116747 err="collector returned no data" ts=2024-06-21T20:33:21.541Z caller=fibrechannel_linux.go:101 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping" ts=2024-06-21T20:33:21.541Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=9.6302e-05 err="collector returned no data" ts=2024-06-21T20:33:21.541Z caller=hwmon_linux.go:440 level=debug collector=hwmon msg="hwmon collector metrics are not available for this system" ts=2024-06-21T20:33:21.541Z caller=collector.go:167 level=debug msg="collector returned no data" name=hwmon duration_seconds=7.0611e-05 err="collector returned no data" ts=2024-06-21T20:33:21.541Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu ts=2024-06-21T20:33:21.541Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io ts=2024-06-21T20:33:21.541Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory ts=2024-06-21T20:33:21.542Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=0.000722049 ts=2024-06-21T20:33:21.543Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.001284556 ts=2024-06-21T20:33:21.543Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded" ts=2024-06-21T20:33:21.543Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=9.2716e-05 err="collector returned no data" ts=2024-06-21T20:33:21.543Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=0.000272085 ts=2024-06-21T20:33:21.543Z caller=nvme_linux.go:56 level=debug collector=nvme msg="nvme statistics not found, skipping" ts=2024-06-21T20:33:21.544Z caller=collector.go:167 level=debug msg="collector returned no data" name=nvme duration_seconds=8.9869e-05 err="collector returned no data" ts=2024-06-21T20:33:21.544Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping" ts=2024-06-21T20:33:21.544Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=0.000108483 err="collector returned no data" ts=2024-06-21T20:33:21.544Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=0.000421419 ts=2024-06-21T20:33:21.545Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=0.000415263 ts=2024-06-21T20:33:21.545Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=3.7784e-05 ts=2024-06-21T20:33:21.545Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=8.8071e-05 ts=2024-06-21T20:33:21.545Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory" ts=2024-06-21T20:33:21.545Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=0.000139228 err="collector returned no data" ts=2024-06-21T20:33:21.545Z caller=rapl_linux.go:77 level=debug collector=rapl msg="Platform doesn't have powercap files present" err="unable to read class/powercap: open /sys/class/powercap: no such file or directory" ts=2024-06-21T20:33:21.545Z caller=collector.go:167 level=debug msg="collector returned no data" name=rapl duration_seconds=8.7256e-05 err="collector returned no data" ts=2024-06-21T20:33:21.546Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.000542391 ts=2024-06-21T20:33:21.546Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=0.000276705 ts=2024-06-21T20:33:21.546Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=3.9723e-05 ts=2024-06-21T20:33:21.546Z caller=schedstat_linux.go:78 level=debug collector=schedstat msg="schedstat file does not exist" ts=2024-06-21T20:33:21.546Z caller=collector.go:167 level=debug msg="collector returned no data" name=schedstat duration_seconds=5.7399e-05 err="collector returned no data" ts=2024-06-21T20:33:21.549Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.003016928 ts=2024-06-21T20:33:21.549Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=2.9824e-05 ts=2024-06-21T20:33:21.550Z caller=watchdog.go:106 level=debug collector=watchdog msg="Could not read watchdog stats" err="failed to list watchdog devices at \"/sys/class/watchdog\": open /sys/class/watchdog: no such file or directory" ts=2024-06-21T20:33:21.550Z caller=collector.go:167 level=debug msg="collector returned no data" name=watchdog duration_seconds=6.325e-05 err="collector returned no data" ts=2024-06-21T20:33:21.550Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.719002001550096e+09 ts=2024-06-21T20:33:21.550Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=0 time_zone=UTC ts=2024-06-21T20:33:21.550Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [kvm-clock tsc] kvm-clock}]" ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=0.00028476 ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=2.035e-06 ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=0.000345937 ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=7.756e-05 ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=3.8736e-05 ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=8.0736e-05 ts=2024-06-21T20:33:21.551Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs ts=2024-06-21T20:33:21.551Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available" ts=2024-06-21T20:33:21.551Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=6.4907e-05 err="collector returned no data" ts=2024-06-21T20:33:21.551Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping" ts=2024-06-21T20:33:21.551Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=3.1584e-05 err="collector returned no data" ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=0.00010089 ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000339368 ts=2024-06-21T20:33:21.552Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=0.00015717 ts=2024-06-21T20:33:21.553Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.001230999 ts=2024-06-21T20:33:21.553Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0 ts=2024-06-21T20:33:21.553Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0 ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1 ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1 ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2 ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2 ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3 ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3 ts=2024-06-21T20:33:21.554Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=1.5636e-05 ts=2024-06-21T20:33:21.554Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net ts=2024-06-21T20:33:21.554Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=5.8544e-05 err="collector returned no data" ts=2024-06-21T20:33:21.554Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=1.7544e-05 ts=2024-06-21T20:33:21.555Z caller=mdadm_linux.go:115 level=debug collector=mdadm msg="Not collecting mdstat, file does not exist" file=/proc ts=2024-06-21T20:33:21.555Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory" ts=2024-06-21T20:33:21.555Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=5.0574e-05 err="collector returned no data" ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4 ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4 ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5 ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5 ts=2024-06-21T20:33:21.555Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.002288325 ts=2024-06-21T20:33:21.556Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=0.001697304 ts=2024-06-21T20:33:21.556Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type" ts=2024-06-21T20:33:21.556Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.001886767 ts=2024-06-21T20:33:21.558Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=0.004203471 ts=2024-06-21T20:33:21.559Z caller=collector.go:167 level=debug msg="collector returned no data" name=mdadm duration_seconds=7.3392e-05 err="collector returned no data" ts=2024-06-21T20:33:21.559Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=0.004094869 ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service ts=2024-06-21T20:33:21.560Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=0.009668599 ts=2024-06-21T20:33:34.805Z caller=node_exporter.go:79 level=debug msg="collect query:" filters="unsupported value type" ts=2024-06-21T20:33:34.806Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=3.5575e-05 ts=2024-06-21T20:33:34.806Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping" ts=2024-06-21T20:33:34.806Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=0.000175007 err="collector returned no data" ts=2024-06-21T20:33:34.806Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=6.5867e-05 ts=2024-06-21T20:33:34.807Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000210497 ts=2024-06-21T20:33:34.807Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs ts=2024-06-21T20:33:34.807Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=0.003866769 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.000544537 ts=2024-06-21T20:33:34.808Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5 ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5 ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.003408076 ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=5.085e-05 ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=4.6219e-05 ts=2024-06-21T20:33:34.808Z caller=mdadm_linux.go:115 level=debug collector=mdadm msg="Not collecting mdstat, file does not exist" file=/proc ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=mdadm duration_seconds=0.003146689 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=5.3531e-05 ts=2024-06-21T20:33:34.808Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=0.003082385 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type" ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.003121106 ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=3.816e-06 ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=2.3564e-05 ts=2024-06-21T20:33:34.808Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=0.003065661 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=3.749e-06 ts=2024-06-21T20:33:34.808Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=0.68 ts=2024-06-21T20:33:34.811Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=0.36 ts=2024-06-21T20:33:34.811Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=0.15 ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=0.003108427 ts=2024-06-21T20:33:34.808Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=0.003101571 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=5.2847e-05 ts=2024-06-21T20:33:34.808Z caller=nvme_linux.go:56 level=debug collector=nvme msg="nvme statistics not found, skipping" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=nvme duration_seconds=0.003043684 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=0.003044284 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=fibrechannel_linux.go:101 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=0.003022354 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=hwmon_linux.go:440 level=debug collector=hwmon msg="hwmon collector metrics are not available for this system" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=hwmon duration_seconds=0.003022138 err="collector returned no data" ts=2024-06-21T20:33:34.808Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu ts=2024-06-21T20:33:34.811Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io ts=2024-06-21T20:33:34.811Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=0.003136247 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.000272793 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=9.795e-05 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=8.6614e-05 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=6.062e-06 ts=2024-06-21T20:33:34.809Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=0.00269225 err="collector returned no data" ts=2024-06-21T20:33:34.809Z caller=rapl_linux.go:77 level=debug collector=rapl msg="Platform doesn't have powercap files present" err="unable to read class/powercap: open /sys/class/powercap: no such file or directory" ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=rapl duration_seconds=0.002685124 err="collector returned no data" ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.000134776 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=0.000104789 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=1.2511e-05 ts=2024-06-21T20:33:34.809Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory" ts=2024-06-21T20:33:34.812Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=0.002529992 err="collector returned no data" ts=2024-06-21T20:33:34.809Z caller=watchdog.go:106 level=debug collector=watchdog msg="Could not read watchdog stats" err="failed to list watchdog devices at \"/sys/class/watchdog\": open /sys/class/watchdog: no such file or directory" ts=2024-06-21T20:33:34.812Z caller=collector.go:167 level=debug msg="collector returned no data" name=watchdog duration_seconds=0.002531731 err="collector returned no data" ts=2024-06-21T20:33:34.809Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.719002014809634e+09 ts=2024-06-21T20:33:34.812Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=0 time_zone=UTC ts=2024-06-21T20:33:34.812Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [kvm-clock tsc] kvm-clock}]" ts=2024-06-21T20:33:34.812Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=0.002609163 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=9.66e-07 ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=1.2051e-05 ts=2024-06-21T20:33:34.809Z caller=schedstat_linux.go:78 level=debug collector=schedstat msg="schedstat file does not exist" ts=2024-06-21T20:33:34.812Z caller=collector.go:167 level=debug msg="collector returned no data" name=schedstat duration_seconds=0.002609169 err="collector returned no data" ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.000960856 ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=4.928e-06 ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=3.48e-06 ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=3.6102e-05 ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=0.000113626 ts=2024-06-21T20:33:34.811Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service ts=2024-06-21T20:33:35.814Z caller=filesystem_linux.go:173 level=debug collector=filesystem msg="Mount point timed out, it is being labeled as stuck and will not be monitored" mountpoint=/media/test1 ts=2024-06-21T20:33:45.033Z caller=filesystem_linux.go:131 level=debug collector=filesystem msg="Mount point has recovered, monitoring will resume" mountpoint=/media/test1 ts=2024-06-21T20:33:45.033Z caller=filesystem_linux.go:138 level=debug collector=filesystem msg="Error on statfs() system call" rootfs=/media/test1 err="host is down" ts=2024-06-21T20:33:45.033Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=10.223095789 ```

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

  1. Set up two VMs. The first one will mount the file system of the second one.
  2. Configure samba on the second VM.
  3. Configure fstab on the first VM.
    //192.168.0.1/debian /media/test1    cifs    defaults,_netdev,username=debian,password=debian    0 0
  4. mkdir /media/test1 on the first VM.
  5. mount /media/test1 on the first VM.
  6. Start node exporter on the first VM.
  7. Stop the second VM.
  8. Run time curl -s http://localhost:9100/metrics | grep node_scrape_collector_duration_seconds.

If you don't like cifs, replace it with sshfs.

What did you expect to see?

The exporter should respond in a about 1 second. node_scrape_collector_duration_seconds{collector="filesystem"} 1.20955266

What did you see instead?

The exporter may freeze for an unknown amount of time, but may respond in 10 seconds.

chengjoey commented 1 week ago

look issue #2903, this seems to be the expected behavior

vitoyucepi commented 1 week ago

I'm not sure if this behavior is expected, because in my test case the result can vary from 10.2s to an unknown amount of time, which in both cases is longer than the default timeout for Prometheus.

I discovered this problem after another network outage on my VM. During the outage, local Prometheus didn't collect any node exporter metrics because the scrape_timeout is 10s by default.

chengjoey commented 1 week ago

I'm not sure if this behavior is expected, because in my test case the result can vary from 10.2s to an unknown amount of time, which in both cases is longer than the default timeout for Prometheus.

I discovered this problem after another network outage on my VM. During the outage, local Prometheus didn't collect any node exporter metrics because the scrape_timeout is 10s by default.

see #3063, I tried to fix the timeout not working issue, and #2903 was reverted because it was necessary to ensure that it could be removed from stuckMounts after successful execution.

vitoyucepi commented 1 week ago

f7b34139 works for me.