search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.29k stars 2.37k forks source link

Node_exporter doesn't match data in pod vs host (--collector.mountstats ). #893

Closed Davidchinacloud closed 6 years ago

Davidchinacloud commented 6 years ago

Hello,

I've set up prometheus with node_exporter on k8s pod using CentOS 7.4 .
I find there is a difference between the mountstats of POD and the mountstats of host. I want to know how to use host's mountstats data in pod:

################### node-exporter.yaml########################

apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter labels: app: node-exporter namespace: kube-system spec: selector: matchLabels: app: node-exporter template: metadata: labels: app: node-exporter name: node-exporter spec: containers:

Host operating system: output of uname -a

Linux 4.4.121-1.el7.elrepo.x86_64 #1 SMP Sun Mar 11 14:33:50 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter --version

node_exporter, version 0.15.2 (branch: , revision: ) build user: copr build date: 20180108-08:17:01 go version: go1.8.3

node_exporter command line flags

/bin/node_exporter --web.listen-address=:9110 --collector.nfs --collector.ksmd --collector.mountstats --collector.systemd --collector.tcpstat

################ run in pod ############################################### [root@node193 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 278ff88e921f quay.io/prometheus/node-exporter@sha256:0344f9a4511426f2ef71ebba20466cb8d0e2104aab8785a32777091bee26cbb7 "/bin/node_exporter -" 6 hours ago Up 6 hours k8s_node-exporter_node-exporter-ngdkg_kube-system_ab0b9296-3d26-11e8-a753-fa163e641f24_0

[root@node193 ~]# docker exec -it 278ff88e921f sh / $ cat /proc/self/mountstats

device /dev/mapper/docker-253:0-68044239-3ab93587c1fb64e769e4b86da30ac669272370e370cdaa48fefc695e684753bd mounted on / with fstype xfs device proc mounted on /proc with fstype proc device tmpfs mounted on /dev with fstype tmpfs device devpts mounted on /dev/pts with fstype devpts device sysfs mounted on /sys with fstype sysfs device tmpfs mounted on /sys/fs/cgroup with fstype tmpfs device cgroup mounted on /sys/fs/cgroup/systemd with fstype cgroup device cgroup mounted on /sys/fs/cgroup/cpu,cpuacct with fstype cgroup device cgroup mounted on /sys/fs/cgroup/hugetlb with fstype cgroup device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup device cgroup mounted on /sys/fs/cgroup/net_cls,net_prio with fstype cgroup device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup device mqueue mounted on /dev/mqueue with fstype mqueue device /dev/mapper/vg_sys-root mounted on /dev/termination-log with fstype xfs device /dev/mapper/vg_sys-root mounted on /etc/resolv.conf with fstype xfs device /dev/mapper/vg_sys-root mounted on /etc/hostname with fstype xfs device /dev/mapper/vg_sys-root mounted on /etc/hosts with fstype xfs device shm mounted on /dev/shm with fstype tmpfs device tmpfs mounted on /var/run/secrets/kubernetes.io/serviceaccount with fstype tmpfs

###################### in host ############################################

[root@node193 ~]# cat /proc/self/mountstats

device sysfs mounted on /sys with fstype sysfs device proc mounted on /proc with fstype proc device devtmpfs mounted on /dev with fstype devtmpfs device securityfs mounted on /sys/kernel/security with fstype securityfs device tmpfs mounted on /dev/shm with fstype tmpfs device devpts mounted on /dev/pts with fstype devpts device tmpfs mounted on /run with fstype tmpfs device tmpfs mounted on /sys/fs/cgroup with fstype tmpfs device cgroup mounted on /sys/fs/cgroup/systemd with fstype cgroup device pstore mounted on /sys/fs/pstore with fstype pstore device cgroup mounted on /sys/fs/cgroup/cpu,cpuacct with fstype cgroup device cgroup mounted on /sys/fs/cgroup/hugetlb with fstype cgroup device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup device cgroup mounted on /sys/fs/cgroup/net_cls,net_prio with fstype cgroup device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup device configfs mounted on /sys/kernel/config with fstype configfs device /dev/mapper/vg_sys-root mounted on / with fstype xfs device systemd-1 mounted on /proc/sys/fs/binfmt_misc with fstype autofs device mqueue mounted on /dev/mqueue with fstype mqueue device debugfs mounted on /sys/kernel/debug with fstype debugfs device hugetlbfs mounted on /dev/hugepages with fstype hugetlbfs device nfsd mounted on /proc/fs/nfsd with fstype nfsd device /dev/vda1 mounted on /boot with fstype xfs device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs device tmpfs mounted on /var/lib/kubelet/pods/428ee5c3-3bd2-11e8-8bc7-fa163e641f24/volumes/kubernetes.io~secret/default-token-8j762 with fstype tmpfs device tmpfs mounted on /var/lib/kubelet/pods/3a295b46-3bd2-11e8-8bc7-fa163e641f24/volumes/kubernetes.io~secret/default-token-dgv5p with fstype tmpfs device tmpfs mounted on /var/lib/kubelet/pods/c797d6b6-3c88-11e8-a753-fa163e641f24/volumes/kubernetes.io~secret/default-token-8j762 with fstype tmpfs device tmpfs mounted on /var/lib/kubelet/pods/cc8dcc92-3bd5-11e8-8bc7-fa163e641f24/volumes/kubernetes.io~secret/default-token-8j762 with fstype tmpfs device tmpfs mounted on /var/lib/kubelet/pods/5a454bd7-3bd7-11e8-8bc7-fa163e641f24/volumes/kubernetes.io~secret/flannel-token-pz95v with fstype tmpfs device 172.16.74.101:/home/nfs/rooot/kube-system-prometheus-data-pvc-73d8162d-3bd2-11e8-8bc7-fa163e641f24 mounted on /var/lib/kubelet/pods/c797d6b6-3c88-11e8-a753-fa163e641f24/volumes/kubernetes.io~nfs/pvc-73d8162d-3bd2-11e8-8bc7-fa163e641f24 with fstype nfs4 statvers=1.1 opts: rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.71.193,local_lock=none age: 25911 impl_id: name='',domain='',date='0,0' caps: caps=0x3ffdf,wtmult=512,dtsize=32768,bsize=0,namlen=255 nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x800,acl=0x3,sessions,pnfs=not configured sec: flavor=1,pseudoflavor=1 events: 4416 23014 38 2727 1368 43 28883 147024 2816 117 0 7103 1806 22 2674 2670 0 2674 0 22 4404 0 0 0 0 0 0 bytes: 732410479 587310459 0 0 895926667 597856227 218756 145979 RPC iostats version: 1.0 p/v: 100003/4 (nfs) xprt: tcp 811 0 1 0 6 16173 16172 0 44239 0 31 692 11897 per-op statistics NULL: 0 0 0 0 0 0 0 0 READ: 3704 3704 0 725984 896311952 81 143864 143998 WRITE: 2718 2718 0 598443360 478368 11785 71968 83792 COMMIT: 15 15 0 2700 1560 0 970 970 OPEN: 44 44 0 12552 15664 0 402 403 OPEN_CONFIRM: 0 0 0 0 0 0 0 0 OPEN_NOATTR: 71 71 0 17608 25412 0 60 61 OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0 CLOSE: 98 98 0 19992 17248 1 46 48 SETATTR: 24 24 0 5440 6528 0 25 26 FSINFO: 1 1 0 168 152 0 0 0 RENEW: 0 0 0 0 0 0 0 0 SETCLIENTID: 0 0 0 0 0 0 0 0 SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0 LOCK: 0 0 0 0 0 0 0 0 LOCKT: 0 0 0 0 0 0 0 0 LOCKU: 0 0 0 0 0 0 0 0 ACCESS: 4406 4406 0 828316 740208 112 951 5423 GETATTR: 4417 4417 0 795048 1095400 90 1132 5970 LOOKUP: 67 67 0 13736 12944 0 492 495 LOOKUP_ROOT: 0 0 0 0 0 0 0 0 REMOVE: 33 33 0 6088 3708 0 179 180 RENAME: 19 19 0 4840 2888 0 21 21 LINK: 1 1 0 240 300 0 0 0 SYMLINK: 0 0 0 0 0 0 0 0 CREATE: 10 10 0 2360 3320 0 31 32 PATHCONF: 1 1 0 164 116 0 0 0 STATFS: 7 7 0 1288 1120 0 231 232 READLINK: 0 0 0 0 0 0 0 0 READDIR: 455 455 0 92820 740944 2 153 1451 SERVER_CAPS: 2 2 0 344 328 0 0 0 DELEGRETURN: 53 53 0 10600 8480 305 18 325 GETACL: 0 0 0 0 0 0 0 0 SETACL: 0 0 0 0 0 0 0 0 FS_LOCATIONS: 0 0 0 0 0 0 0 0 RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0 SECINFO: 0 0 0 0 0 0 0 0 FSID_PRESENT: 0 0 0 0 0 0 0 0 EXCHANGE_ID: 0 0 0 0 0 0 0 0 CREATE_SESSION: 0 0 0 0 0 0 0 0 DESTROY_SESSION: 0 0 0 0 0 0 0 0 SEQUENCE: 0 0 0 0 0 0 0 0 GET_LEASE_TIME: 0 0 0 0 0 0 0 0 RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0 LAYOUTGET: 0 0 0 0 0 0 0 0 GETDEVICEINFO: 0 0 0 0 0 0 0 0 LAYOUTCOMMIT: 0 0 0 0 0 0 0 0 LAYOUTRETURN: 0 0 0 0 0 0 0 0 SECINFO_NO_NAME: 0 0 0 0 0 0 0 0 TEST_STATEID: 0 0 0 0 0 0 0 0 FREE_STATEID: 0 0 0 0 0 0 0 0 GETDEVICELIST: 0 0 0 0 0 0 0 0 BIND_CONN_TO_SESSION: 0 0 0 0 0 0 0 0 DESTROY_CLIENTID: 0 0 0 0 0 0 0 0 SEEK: 0 0 0 0 0 0 0 0 ALLOCATE: 0 0 0 0 0 0 0 0 DEALLOCATE: 0 0 0 0 0 0 0 0 LAYOUTSTATS: 0 0 0 0 0 0 0 0 CLONE: 0 0 0 0 0 0 0 0

device tmpfs mounted on /var/lib/kubelet/pods/ab0b9296-3d26-11e8-a753-fa163e641f24/volumes/kubernetes.io~secret/default-token-8j762 with fstype tmpfs device tmpfs mounted on /run/user/0 with fstype tmpfs

SuperQ commented 6 years ago

GitHub issues should only be used for feature requests and bug reports. For general usage/help/discussions, please refer to one of: