search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.3k stars 2.38k forks source link

node_filesystem_readonly is showing RW, but filesystem is actually RO #591

Closed onorua closed 6 years ago

onorua commented 7 years ago

Host operating system: output of uname -a

Linux lhss-nyc2-prod-15 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:54 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter -version

/ # node_exporter --version
node_exporter, version 0.14.0 (branch: master, revision: 840ba5dcc71a084a3bc63cb6063003c1f94435a6)
  build user:       root@bb6d0678e7f3
  build date:       20170321-12:12:54
  go version:       go1.7.5

Are you running node_exporter in Docker?

  node-exporter:
    container_name: node-exporter
    image: "prom/node-exporter:v0.14.0"
    network_mode: host
    mem_limit: 60M
    restart: always
    command:
      - '-collector.procfs=/host_proc'
      - '-collector.sysfs=/host_sys'
      - '-collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($$|/)"'
    ports:
      - "9100:9100"
    environment:
      - CONSUL_EXPORT_NODE-EXPORTER=9100
    volumes:
      - /proc:/host_proc:ro
      - /sys:/host_sys:ro
      - /:/rootfs

What did you do that produced an error?

Just check filesystem status from prometheus server, node_exporter reported filesystem as RW, but it is RO for real.

/proc/mounts file

cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=2015212k,nr_inodes=503803,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=404668k,mode=755 0 0
/dev/vda1 / ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,noatime,nodiratime 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
/dev/vda1 /var/lib/docker/overlay2 ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
nsfs /run/docker/netns/default nsfs rw 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
overlay /var/lib/docker/overlay2/3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/merged overlay rw,relatime,lowerdir=l/3BG4CPAQ53BVJ6MONFIOQTI6TP:l/CP3RYUIBQXLJH7YN7D65LZ3KDJ:l/GFWF3CBKLNI34UGY7TEULWC4T6:l/HF23JW4HM6LDKAR7C7VW2WFVRT,upperdir=3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/diff,workdir=3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/work 0 0
overlay /var/lib/docker/overlay2/b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/merged overlay rw,relatime,lowerdir=l/XF2NMAJUM74T7U3MGQRKGBUDM6:l/JIRZCYFTPOODFPMYKMTCTIJ5SB:l/PEVBKEUM3JKO4QS4OE7KX6STKX:l/IBENPFOXDDXO4LJ4IZ3QVSUMXB:l/EYQNXNBLIIJFKIOVYHYU5JSOB3:l/ZB2T7R2ANOK7WHREXPDXKIIXEM:l/F5MQQ27KT43F4QGUIA6HSZFD2Z,upperdir=b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/diff,workdir=b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/work 0 0
shm /var/lib/docker/containers/abdd6c4e1bf9e63bf6afab17913d1b7c434d32df0659163a16e6ca4cbfa8b346/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/merged overlay rw,relatime,lowerdir=l/BRXXP3442WG4QT6G27YBYADKEM:l/N2PZTBZJHWLCTTJWBW4WVU3FCT:l/ATVYPEIKPQFUOBIVS3LWIKGTQK:l/2CL5YUBRWWTRD563QX6KMVER74:l/MIK5KHZOJDC2NO7GSZH2S5MDMK:l/HQQ7D35VIVIEJGWJIIW7FP4N7J:l/YAZAP37C5YZDHGKLXQYDT5Q2VW:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/diff,workdir=686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/work 0 0
shm /var/lib/docker/containers/9e1da532483b0632fccf489931520a08f1833eda2b05d6394e29d507a0977c31/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/merged overlay rw,relatime,lowerdir=l/6RP6XWW6BHYW5V4LRFSYJESODU:l/BFJDVRPVBJOLNQQGPCZH6V3B4B:l/PCAVIKPKGVV2R3QC6LVAUHQL5K:l/Z2L2NKG4ZXYK56XLSHDH4RFRXV:l/PZM57C3GVDXGRQS7PK7K4AVUM6:l/QBPTKHSWVJXL7TWHN5OQ2EHNNP:l/S5367OB7IR5VXRDDGODFLQWKX3:l/267KQMWEPLGOWGC67ZENWRZVG5:l/6YQNUXFRJPY72T7NWW2KNQTOYE,upperdir=31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/diff,workdir=31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/work 0 0
shm /var/lib/docker/containers/c7e7945e69fc410589bd20196b87dd1ffc707de2fe8f6f0c177b71e16f6fb029/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged overlay rw,relatime,lowerdir=l/XWEPIUFAPP7ZI7GICWFLWYR2ZV:l/R5PG7LY6KPIUEYEZIQFPM55HZL:l/2YYFFSQILMJNGDI2EY64BA3DCK:l/M6FZJXQE2J3O4QLIF3Z6OV3IYV:l/OAPL6OCFMDPJSCFK5H5OKUZBB2:l/TVOOCBC2GTB3FRR3MMU6NAKDYF:l/HGVPR35DTAXOOIWH74Y4BSOUCK:l/4LMQBXJDIVIK2D5DI3WW473ZXZ,upperdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/diff,workdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/work 0 0
shm /var/lib/docker/containers/dc101ac35f7edc560434c55d9485c00b832c47bdbfa4035f9af37b41e5dd0133/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/e3dd5f9f858e178dd1791ea0ac75873891ebc03a7143ce5282b3f29ead7b9057/merged overlay rw,relatime,lowerdir=l/YBD2K7GJLC6X4AG6SKIVKYBJZ3:l/PNSDAF54YMYHCTAIFBE7XL7HCX:l/H7WX2VXLDYCRUDUKJN7SCMKXOQ:l/METFBHQH7V4JNNR6UVASJOJXPE:l/CSFTHUBNUH3RC4UBURPF54ZGM2:l/7YSPDYBNLWOQ7CWDYOQLV3YZPD:l/562CEQO75XTY6UXU4TX226T535,upperdir=e3dd5f9f858e178dd1791ea0ac75873891ebc03a7143ce5282b3f29ead7b9057/diff,workdir=e3dd5f9f858e178dd1791ea0ac75873891ebc03a7143ce5282b3f29ead7b9057/work 0 0
shm /var/lib/docker/containers/33f02ada6e6e3698742b0eab52982080ab5589567724580703fe8885b00c16f6/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/6bf604f6676449326e15e01a01d021da4fa150f219d3084b95d7b47ceafc3559/merged overlay rw,relatime,lowerdir=l/DHT2VMJI2JSICXSBMAL2AZCONB:l/BADYPTXBCVH6RKYR27EJQBZBVU:l/GDDE23U3ZS5QP7SH7EUCQXKMTS:l/C6MH6I25HMNO5WHZYRJX3HZHNX:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=6bf604f6676449326e15e01a01d021da4fa150f219d3084b95d7b47ceafc3559/diff,workdir=6bf604f6676449326e15e01a01d021da4fa150f219d3084b95d7b47ceafc3559/work 0 0
overlay /var/lib/docker/overlay2/65cf9af585b0f3903429de31488b31f147a14c13a2d12fb11823ed22529cf1a7/merged overlay rw,relatime,lowerdir=l/AEXD42UDHUPIYKSJ332RSVBM6E:l/QFSLFRRXC5NNYDZ2DAJZJFZZ4G:l/BZ7B3JSSLOV7HD54BCTGCRWM2H:l/52NOYFLRTBSHVFZ5NVYATGCY73:l/UIEAFNER25HREXE6OBFWS3UP7B:l/M6MZIHXRZPCJT6M74YXABF3YYA:l/W64RVD2PBLFLTXGBIVXFSV57OS:l/SUSMWOMW5CRCFPV2UBBO2ZOFXL:l/JO4LDPRR4TVFSECKZ7SE5WH5UL:l/MSDV4B2OC3MCUKQM3XP5BR7CIS:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=65cf9af585b0f3903429de31488b31f147a14c13a2d12fb11823ed22529cf1a7/diff,workdir=65cf9af585b0f3903429de31488b31f147a14c13a2d12fb11823ed22529cf1a7/work 0 0
shm /var/lib/docker/containers/57f22cdce9f60c69726e21b29a7ddd4a94e2b03f8d0e58754e879ecb9fe6ca5c/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /var/lib/docker/containers/51b68b7663a0a75ebe96731ca92e7d1ca23bed087021424b0d41c368973a8c6f/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/76c8de59cdc2 nsfs rw 0 0
overlay /var/lib/docker/overlay2/4d0741cf64a57fe38874358ec2e56ec77301f60b74046c5c3b9f5a4140d22b26/merged overlay rw,relatime,lowerdir=l/E26DOIDEWL4JTIVBC5J5YLNAQN:l/VH6ZYNLP6LYYDZ3LJZA7UT3JCY:l/ICPBU5ZTIDXL6FK3I6MN6F5C76:l/WQWT4DPTG3SEHUBNVMLIVI6I3X:l/562CEQO75XTY6UXU4TX226T535,upperdir=4d0741cf64a57fe38874358ec2e56ec77301f60b74046c5c3b9f5a4140d22b26/diff,workdir=4d0741cf64a57fe38874358ec2e56ec77301f60b74046c5c3b9f5a4140d22b26/work 0 0
shm /var/lib/docker/containers/bc74830ec1b4391acf7b7a9ed353366489b629922b6a744ea286a944d0704b02/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/2c65c0a47b5bcf34d8cb21aab01f573e91c7488201a66cbf68c1d7c296c0fa9a/merged overlay rw,relatime,lowerdir=l/7YS3Y5ISIPZH6AEUZQW36Z7C7V:l/WW42JWB6GTBY6GRWMB4QXJYDTY:l/AFG2P5SSBKAIWOIG2AXRNOG75O:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=2c65c0a47b5bcf34d8cb21aab01f573e91c7488201a66cbf68c1d7c296c0fa9a/diff,workdir=2c65c0a47b5bcf34d8cb21aab01f573e91c7488201a66cbf68c1d7c296c0fa9a/work 0 0
shm /var/lib/docker/containers/56d8d2b8292b150ab9d322d50875411547acb7b4076e264a03be11613eccf305/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/f139d78b7661ee11e567c520bd6d51862ffc36851fa4d29f4fe168d0814c542d/merged overlay rw,relatime,lowerdir=l/BE57KFYMA7AKIK3T5C2LRSHY32:l/N5G4SVHKQHRYCUZVHDU3DQKLZT:l/GFWMA4VHY5TBHADKL56B2RE3ED:l/HRDONYOUN37K3MH5CD662F4NXM:l/JAMKPS2YLPWRIN3HOVM36XERV7:l/CF4EK4MKJM3PCS3AS6F6YIME7V:l/DMWX4TQ2HNJAODZTCPU65XDIZ4:l/BDF6LYJR2K3ORC7KH4S5E54JXB:l/GSK45JTGLMDDZSHCBNLWLKWL4P:l/S5HZMUW7QU4G6MHCNMX7R3JAQZ:l/3XCRIB6HY6OQTHWBCBUZGZDS2G:l/2PUJTWPRMQ5VOXXCNSONR4F2SS:l/GOK6L2AE2I6WIHCT3W4T4DLTPM:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=f139d78b7661ee11e567c520bd6d51862ffc36851fa4d29f4fe168d0814c542d/diff,workdir=f139d78b7661ee11e567c520bd6d51862ffc36851fa4d29f4fe168d0814c542d/work 0 0
overlay /var/lib/docker/overlay2/04fee163764e681b89820ff1d298771d3a5739d7dbd0cb386b63752e5aea531c/merged overlay rw,relatime,lowerdir=l/6VLFNU7EQE7GEVGQZHR4IMLRCS:l/P3P2PPZSTM4Z3EERAZPFY5K2SA:l/FZ2VJOZ4NJF5G6WVOXK4M452DM:l/LIBTIUPHBYQ5SRRWLW7XQ4SYXZ:l/MKEPOLZRJULHJNTBIGRHJB3NR7:l/3DNQ5SVC2T252UUCS664WXAQ7M:l/IPT32VF3OVOUXST3EEAXRWVOZP:l/MSPMOOEAWGE2TSCLY2BXT5JO3Y:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=04fee163764e681b89820ff1d298771d3a5739d7dbd0cb386b63752e5aea531c/diff,workdir=04fee163764e681b89820ff1d298771d3a5739d7dbd0cb386b63752e5aea531c/work 0 0
shm /var/lib/docker/containers/6b44bfa202ffce2fce5d6bc1981727269b5dd07526baa5410ebcf2f452c9d73d/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/8ccf77d59af5df50e0743ac9378b003dec25fbc03cba1d77c017c2f69d5c9939/merged overlay rw,relatime,lowerdir=l/2YGDZFBDJDPUD6HJFU57E725G2:l/IBDR27CWL2NL53BM6U6SEUXYIP:l/I6VCBQWMBELZZBYB7OEKQRPUW3:l/K7TLIJJOFUTGTHLN2CBQGBOCDB:l/6RQJWH25GMDFSKVJ7IKXDW54WR:l/PFRHQUFA4GEXASUAU2CR6VRK57:l/MA4C7V5OGKBSJ5KAU4TIFPFEET,upperdir=8ccf77d59af5df50e0743ac9378b003dec25fbc03cba1d77c017c2f69d5c9939/diff,workdir=8ccf77d59af5df50e0743ac9378b003dec25fbc03cba1d77c017c2f69d5c9939/work 0 0
overlay /var/lib/docker/overlay2/77b20d8845ae32ccc609ed80523ec9f4eff6f9f8bcd821c0e9edebdc00bf0f78/merged overlay rw,relatime,lowerdir=l/K3C4QQ6SXKCOCCU6CCDAKXXADB:l/CFMCKHVU24W3ZQLFNHJL62MAFD:l/RLF5XBGRYE6W6OWTJOUDRGPWOM:l/DBK2UXZCY73SKN27GT3XNCAAJD:l/5C2FDNSYQDRTU6AZH3RDIFNXCF:l/JORA2F6B2HXS7XHBZ6VSWZ2ISU:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=77b20d8845ae32ccc609ed80523ec9f4eff6f9f8bcd821c0e9edebdc00bf0f78/diff,workdir=77b20d8845ae32ccc609ed80523ec9f4eff6f9f8bcd821c0e9edebdc00bf0f78/work 0 0
shm /var/lib/docker/containers/dc4ed9e21112485d6ee67ca1f5b64c16732c1581420c1c0b9d5b312f475d727a/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/7e68f15a2311c6b4421b0053e376e9cb1d3e87ee62b5991fb7ad5d6c45477b09/merged overlay rw,relatime,lowerdir=l/7UWLP6IX7NJ4YQB72AYPKJHZRK:l/5OUOAAD2FZAAS2AQUPSW5OVHAC:l/X6YI4PBOGFOJZDPYYCKKKGBOEA:l/XTTQTESPNTYH7TMEQEXCDVHNAM:l/YXUJVF3ZXTUL4YVDC65XNF5HZA:l/2UYGOOL55QR66R6WIXWDB6WY3P:l/KJ4GQPZOPZ6RLEBQPMHHNXDOTF:l/ZAER47HQ5ZF3SRGHJ45GMDHLHF:l/WTDIYWY2DEGN4D4YZZ53LGE2BU:l/RZ3E23FRTIBMEDROXSNNNB57TO:l/V2W5AI2CTZJPOFNL37R6UOQPCE:l/S3QAIREI2HSXGJFHOAP6DCNKLN:l/PCGNUWUWO25NVC6DBBEB4QLNTU:l/OGSKYF446IHCN44EBCNRQX6HRX,upperdir=7e68f15a2311c6b4421b0053e376e9cb1d3e87ee62b5991fb7ad5d6c45477b09/diff,workdir=7e68f15a2311c6b4421b0053e376e9cb1d3e87ee62b5991fb7ad5d6c45477b09/work 0 0
shm /var/lib/docker/containers/cd50af9091b8280c4419f339a7fbcd1542c2958ec93a915f1797943c8275d80a/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/4662af0b733601ce15e9d68692e50b2f492987a29b7ec3a676486e10865da064/merged overlay rw,relatime,lowerdir=l/VYUPKYB4PQHKAPYSSD6AV3IQPS:l/5OUOAAD2FZAAS2AQUPSW5OVHAC:l/X6YI4PBOGFOJZDPYYCKKKGBOEA:l/XTTQTESPNTYH7TMEQEXCDVHNAM:l/YXUJVF3ZXTUL4YVDC65XNF5HZA:l/2UYGOOL55QR66R6WIXWDB6WY3P:l/KJ4GQPZOPZ6RLEBQPMHHNXDOTF:l/ZAER47HQ5ZF3SRGHJ45GMDHLHF:l/WTDIYWY2DEGN4D4YZZ53LGE2BU:l/RZ3E23FRTIBMEDROXSNNNB57TO:l/V2W5AI2CTZJPOFNL37R6UOQPCE:l/S3QAIREI2HSXGJFHOAP6DCNKLN:l/PCGNUWUWO25NVC6DBBEB4QLNTU:l/OGSKYF446IHCN44EBCNRQX6HRX,upperdir=4662af0b733601ce15e9d68692e50b2f492987a29b7ec3a676486e10865da064/diff,workdir=4662af0b733601ce15e9d68692e50b2f492987a29b7ec3a676486e10865da064/work 0 0
shm /var/lib/docker/containers/d396e24736abdc0ae329e3308a2b3165f30a0bb531fec13aeb1e7bb796c1d529/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/1099a380b47598260d4005613e6333175d759b11f1a6b04fc48b2a31104a28cc/merged overlay rw,relatime,lowerdir=l/673A437C76AY3YW3WSMZXIJEBM:l/SWMKAAU2KKRZY6HSJCJ6QUO6M6:l/755IX5QKEAJOOIIU6NT43UY26D:l/NOEJE7GJIAYBFVXNQPEPBENEPG:l/H2P6RE5T4YI5SEXSMP34T2F7FB:l/562CEQO75XTY6UXU4TX226T535,upperdir=1099a380b47598260d4005613e6333175d759b11f1a6b04fc48b2a31104a28cc/diff,workdir=1099a380b47598260d4005613e6333175d759b11f1a6b04fc48b2a31104a28cc/work 0 0
shm /var/lib/docker/containers/af6beeccb90fa85eb516c96438cd2f49e82eb7a4a92ae47411d885b29655d847/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/5f6bda2aa599 nsfs rw 0 0
overlay /var/lib/docker/overlay2/5174be1b43da19e5da75a9a3679fa99cf393e89372c484cc845a91e0c6489d61/merged overlay rw,relatime,lowerdir=l/6ABO6S6RSNOO7HR7CEY55XGASR:l/63DUYF2VS4IKNYQ54LNF2KHJBY:l/6YLX6HSZ6FCQL7GWVPD2WLQCWY:l/6DKENABXIBZC25DZVZC5F4F6N4:l/LHVECFQKV6ZXASUAZPNSHHETHB,upperdir=5174be1b43da19e5da75a9a3679fa99cf393e89372c484cc845a91e0c6489d61/diff,workdir=5174be1b43da19e5da75a9a3679fa99cf393e89372c484cc845a91e0c6489d61/work 0 0
overlay /var/lib/docker/overlay2/29f186fd8dea22da96b841c9335d6f6c7359a6b525dcfc0cb9f730103a464d90/merged overlay rw,relatime,lowerdir=l/MG2H2WJTU6DNBZUVOJZH5MLTGQ:l/P5BU3TFKZGQMSGIHTLHBCNOY5G:l/YYAKJSWIVFTGAJZA2YQ43QNIBR:l/IUVVROLNZNUVX5PYBUBTQ3PNZT:l/TZQ655KGNSCJ5GATH676U3PVP2:l/JHNROZ5V5WMFLC43CTKCL55QEY:l/MA4C7V5OGKBSJ5KAU4TIFPFEET,upperdir=29f186fd8dea22da96b841c9335d6f6c7359a6b525dcfc0cb9f730103a464d90/diff,workdir=29f186fd8dea22da96b841c9335d6f6c7359a6b525dcfc0cb9f730103a464d90/work 0 0
shm /var/lib/docker/containers/a930efae2d1052337ef38fb281291f057626774ed9592355b824c481722b5e30/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/1f0be0883304cfa77754501f6a6f3a6385457ed018dd0bcf41e8461a401e1a5b/merged overlay rw,relatime,lowerdir=l/LY42C4Y4VGNTIGVS7PLZGCIW4A:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=1f0be0883304cfa77754501f6a6f3a6385457ed018dd0bcf41e8461a401e1a5b/diff,workdir=1f0be0883304cfa77754501f6a6f3a6385457ed018dd0bcf41e8461a401e1a5b/work 0 0
nsfs /run/docker/netns/afe124c50f0d nsfs rw 0 0
shm /var/lib/docker/containers/107e222bdca4262c7693545d6ae5d726eb6d79655fe4137ea0f9d6dfb477c418/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/29d06c0e2b5175420bfd217fda80445eb0184a191ac534015b510f87b6f8946e/merged overlay rw,relatime,lowerdir=l/U6FKGZDAFB4AEK76NGGRD4HL3X:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=29d06c0e2b5175420bfd217fda80445eb0184a191ac534015b510f87b6f8946e/diff,workdir=29d06c0e2b5175420bfd217fda80445eb0184a191ac534015b510f87b6f8946e/work 0 0
overlay /var/lib/docker/overlay2/40b152b91022c658f412c9f972667b4e2857836a64af7ac9437cb8ceb2f53491/merged overlay rw,relatime,lowerdir=l/GHZOCRA76KI66UESTMRDOG5BMW:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=40b152b91022c658f412c9f972667b4e2857836a64af7ac9437cb8ceb2f53491/diff,workdir=40b152b91022c658f412c9f972667b4e2857836a64af7ac9437cb8ceb2f53491/work 0 0
shm /var/lib/docker/containers/e4123df790b1461db62dbdb72fbe5612274e62c5922e6072053e5e2fba92421d/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /var/lib/docker/containers/5d9a8a4625f3c70d0761a5bb74bd8632333f689394038b278af525608fbabe70/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/172e701a8234 nsfs rw 0 0
overlay /var/lib/docker/overlay2/12002438dbb35d69e53faeeee0a8afaf25734d150fbfa05f5eea2f25ac9f02ce/merged overlay rw,relatime,lowerdir=l/LZDWILO3K226HFFC4UKIGH3ETE:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=12002438dbb35d69e53faeeee0a8afaf25734d150fbfa05f5eea2f25ac9f02ce/diff,workdir=12002438dbb35d69e53faeeee0a8afaf25734d150fbfa05f5eea2f25ac9f02ce/work 0 0
shm /var/lib/docker/containers/4b820a0a8bbee46507800b8b9d72d14d93c55037aeae5e53cd9084e2a761577b/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/648777e6cbc1 nsfs rw 0 0
overlay /var/lib/docker/overlay2/88210d1097537c404ee15b6443fe84feca58f570fb6a6461dea7def1aab6cfa0/merged overlay rw,relatime,lowerdir=l/X67DZKT3SPH6GIKBW25OA7AIJ2:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=88210d1097537c404ee15b6443fe84feca58f570fb6a6461dea7def1aab6cfa0/diff,workdir=88210d1097537c404ee15b6443fe84feca58f570fb6a6461dea7def1aab6cfa0/work 0 0
nsfs /run/docker/netns/51eb96785424 nsfs rw 0 0
shm /var/lib/docker/containers/5d375a99310bc125aee862f4ff154c13052c7d7f7ddd4d5a4a4ff49e4296261a/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/7072177a27b5 nsfs rw 0 0
nsfs /run/docker/netns/4e505f39baa3 nsfs rw 0 0
overlay /var/lib/docker/overlay2/279a9f48307dceec2985e66009ca08cf0a859e2a3cdc6a105ff22ece46cb6743/merged overlay rw,relatime,lowerdir=l/ASGWHWGYCJDIDH5WCLHYRVOB3U:l/SSBYLJRQCOQRM6G3Z6AZRPXDIY:l/Z4XANGU5UBPAE2BN6XRLDWEVJM:l/6JFFALMYSXPD3EPKW3JDFJY5KE:l/JTA2PPOT6Y6EXFBWZRLW3DG2ZF:l/MWWMO6PQMRMZOYTYBQKGDL5TU5:l/6WSQB2TPTZKHYZHPWO3GZWIXCD:l/MGDALBCITEZTR5QYSLEM45CFQJ:l/57NSTC2PO3HVX7ZBNL2HHTMYQA:l/WJRGPI7TJK5XPSIS4OWNSFNFRD:l/NYIKCPYM2FG4QW7A4ZLPYXXAGU:l/6FDNL6NCIFPVKCEE2KOD2MFQ5F:l/GXWTDPHDNPRRNXRB5OKK2JML7R:l/NAQEOYNTO2WORLCGB2Z4LTI4BT:l/IXMATN5H64UHH26DXGAUJBRS3H,upperdir=279a9f48307dceec2985e66009ca08cf0a859e2a3cdc6a105ff22ece46cb6743/diff,workdir=279a9f48307dceec2985e66009ca08cf0a859e2a3cdc6a105ff22ece46cb6743/work 0 0
shm /var/lib/docker/containers/627a298efe64cf73d4aba3685ebd2f217b59c6309f1a03416715ba5464f4f272/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/9cf4b56137f8 nsfs rw 0 0
overlay /var/lib/docker/overlay2/956b3bc1da9859fc131264accbdb10f06b7a1d1495b048a8d753b15ebbb697ff/merged overlay rw,relatime,lowerdir=l/AVEK7TFPLJSI7HXGS2P3UYZRBY:l/HLANIVF7ZYWRWYGDDLVU6VZBR5:l/G6IB4BYW6NQ54FIDJ7U4QVAOYN:l/S34BJ25WX4AB5HRBT2JQJL5CZY:l/T2EBQ6PUKNRNQGN3OKJEKJI2L7:l/OCHL7WYND3NVJAGBF4PHOKMB6U:l/PRA2HVVNG72PMXSTDZDCPGMYQF:l/QWIXKB2AI6GY64MHEVD7FW2ETD:l/A2POFG2F4JUSU4UJ44TB4NHDAL:l/SIZ5JFD2VZNGIED2LQALDALDAR:l/I7YX3US37NXNIVOBM6GL7ZXVRQ:l/GPHJ357FFESHMQVXWQRA2YARQG:l/MICHTM542NQSUSONQX7DR3OANI:l/XDL4GVT2H2F2HW77RZRNMV5SSF:l/VD7ORKW6N5HGNYK662NM62UGUZ:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=956b3bc1da9859fc131264accbdb10f06b7a1d1495b048a8d753b15ebbb697ff/diff,workdir=956b3bc1da9859fc131264accbdb10f06b7a1d1495b048a8d753b15ebbb697ff/work 0 0
shm /var/lib/docker/containers/dfff0fbe06984a5dfd6c331ab0dd5670aceda7bb68609b3bb61f836139b0d932/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/50fc677bce4b127adce650bf19f1aa18b0d9b887a76147fd9dd0b5aebb7f77a7/merged overlay rw,relatime,lowerdir=l/VCBIMGEMIODEP6JP4ON5WUPXRR:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=50fc677bce4b127adce650bf19f1aa18b0d9b887a76147fd9dd0b5aebb7f77a7/diff,workdir=50fc677bce4b127adce650bf19f1aa18b0d9b887a76147fd9dd0b5aebb7f77a7/work 0 0
shm /var/lib/docker/containers/a6bff365eb62526b9bddf45888e88c634724831c7b3f0a36ad40deabecfff659/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/f90fd68a05a2 nsfs rw 0 0
nsfs /run/docker/netns/45741e5fc002 nsfs rw 0 0
overlay /var/lib/docker/overlay2/e90727dcdf49455c8e73050690353a9661ccda258507179423a2fd5d440dd97c/merged overlay rw,relatime,lowerdir=l/TD6MO4LPIHXW7S55HV5ASBIGUF:l/Z74INQS3QUKS33BH5SWFKYECAL:l/QWYJ33TPVNMM3G43GEYMNVP6WP:l/JUCWLPSGVW27SSNZRJB53SFZUE,upperdir=e90727dcdf49455c8e73050690353a9661ccda258507179423a2fd5d440dd97c/diff,workdir=e90727dcdf49455c8e73050690353a9661ccda258507179423a2fd5d440dd97c/work 0 0
overlay /var/lib/docker/overlay2/a8e76f39b139bb076fc98033a4c4cbace5ec17c53f2c55e78c32db23f72e4b01/merged overlay rw,relatime,lowerdir=l/Z53CAPAOA5W5XTPWQWUQ77WRXO:l/QNAR5EZDEO34TC7VH7BGOU6XJM:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=a8e76f39b139bb076fc98033a4c4cbace5ec17c53f2c55e78c32db23f72e4b01/diff,workdir=a8e76f39b139bb076fc98033a4c4cbace5ec17c53f2c55e78c32db23f72e4b01/work 0 0
shm /var/lib/docker/containers/fe57216cd6b7716094531d33a2bf6e78b58d7bb04431ca75fbcb437cac86553b/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /var/lib/docker/containers/8fb3b3a02af0513c3a782e133b9f3f0bd150d9a888c55b75c200978c741b703e/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/81d64a728479fb2fbdf694ee3b03d73e9182236d5bf682a247ad73580a39b08f/merged overlay rw,relatime,lowerdir=l/MJNF24CSO34SH5UVCWZAKAPOCZ:l/RJCFDHYQJ3VOU3EUV24R6K2KR7:l/GUSD7J7VXXSSA4NWYB6NB7JEMU:l/DJHNAAJXCFD6LXATCMO7ZWXJCT:l/XOC6OYPNXLOXWGA3KKV3AMEA72:l/PBMUXCWHVZXUJDX6VMDG6CVT45:l/WU53ZTW7IE4C5GO6O2VBMPGCV3:l/PTAGFOZNVDCAV5EJRTLG3YUXBH:l/3LG6WXCLMOHSKIUZUALCG7CF5Z:l/HSU3GYPQTY442LQPVQGNQ65WXX:l/HV4GPUA5HBWVRKDLEUFTJMRE5U:l/RRYIYPAYVEKT2WCF2SJL3Q3RN7:l/AIEF3JN342NCNOKS2ZEMOZARP4:l/MA4C7V5OGKBSJ5KAU4TIFPFEET,upperdir=81d64a728479fb2fbdf694ee3b03d73e9182236d5bf682a247ad73580a39b08f/diff,workdir=81d64a728479fb2fbdf694ee3b03d73e9182236d5bf682a247ad73580a39b08f/work 0 0
shm /var/lib/docker/containers/fee24d58563a7c0a7afb06cab6488e50d93061dee4d1b2248422c61b76216113/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/653cf6c5e5ce5d2b7eeccb075a9f57bebf974306a91592fb974878f48b38e4a2/merged overlay rw,relatime,lowerdir=l/7RKQVSWIVA4RJGJTDNZS3R7IRH:l/KDNIKWLCPVZT4UMXE2WLA374MQ:l/VT34J2GBQ7LJGZ2IN5DVVIBPYM:l/3LN2PBUYF4F35NKM36IH2E7Y2R:l/AUUYD2YIKWKOILH4X4MNSC5QPE:l/OUMMKWR6IVYYELFDAVD6ZNDVE3:l/WLHKWWMNTFYYQXIANQ5HQLVPDU:l/YGSK7FEL6OZSVAV77EDL6Q2WPG:l/A37U373A235DJLZAN4JPDWCRXD:l/RNQ7SL6ADKOFBIKF66QYB62YTK:l/2T6B32DW5M6TUAHKRHNWYH42MY:l/BGGVNYKWWZE7OF3YVSAAM2V33L:l/CO3MPV4YDA6LCIGA4O4H6T3LAZ:l/FT7PNF3YEAJR7Z5KHJ4S7HOXUF,upperdir=653cf6c5e5ce5d2b7eeccb075a9f57bebf974306a91592fb974878f48b38e4a2/diff,workdir=653cf6c5e5ce5d2b7eeccb075a9f57bebf974306a91592fb974878f48b38e4a2/work 0 0
shm /var/lib/docker/containers/75a22b1b2a0a902b1b6cd59a0c76d7066be30ba9933e434207fdc9439913dc72/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/3d66e96273b9f51ce722743c2b2ce2853e0999fcefd54244d3d6897af13fcd2e/merged overlay rw,relatime,lowerdir=l/PFU36Z4NEVIB355LBVMMLX63L3:l/HBTQLLLHMFWRMMFPABPCBWDLNT:l/J4XQW54NS4OTKB7ZZHTI6O2P6B:l/TD33P3JEZUOIR2F2XBMLEPHBVZ:l/AA4DUZUREEHL2SQB3NPUIEEAVJ:l/SDN7PQWNQTYUZWPANFQV5RGFAZ:l/UBIE2GBEVPICZNMJD5HWRESHWW:l/HFOZZ3M3B5TECZEHYZJ4ZSQJ4E:l/HU7XQNH4ZL6NBTHSVVUTDB4HCX:l/67ZPF2ICP2VKW6IPJKSCBVKMMV:l/5YXZSCZROXZ2HKFXRN43TXU6RV:l/FL6LD7IIQF2KDQVG535GWTXRTG:l/XWGFJN4XFHC2XKDHW6DA77D44C:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=3d66e96273b9f51ce722743c2b2ce2853e0999fcefd54244d3d6897af13fcd2e/diff,workdir=3d66e96273b9f51ce722743c2b2ce2853e0999fcefd54244d3d6897af13fcd2e/work 0 0
shm /var/lib/docker/containers/bcf3097df526ff65237f29a50ebdf4528ee2727fec1ecc4cefd36e8d16c97e02/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/0927d561f4b49ab231f4e0b23fba2cb5810022df0bca01541e5b5fb5db56958a/merged overlay rw,relatime,lowerdir=l/OA5VYUBHW4ON3TEOEV3NGPST6P:l/PTCUOG5TPCGBDRWLNAIZSIBNAP:l/KZVZKNTLAQHJXYEDZIZRJU6AS6:l/G77BPTXLPHEBWKWMGE327PT2P5:l/DN4IEJ3WSIEUMGGRNHZ2B2LVAC:l/AXGCS4BJWFNPY3LEU644XXG2FP:l/A6AC3F4JO5XZ64RYJMR5AYOYUC:l/BIMN7U245V6ZJBGKAIJIPRHZC7,upperdir=0927d561f4b49ab231f4e0b23fba2cb5810022df0bca01541e5b5fb5db56958a/diff,workdir=0927d561f4b49ab231f4e0b23fba2cb5810022df0bca01541e5b5fb5db56958a/work 0 0
shm /var/lib/docker/containers/7b1909e0a9a3d63669d61dd65c92b8556613e52255b92241d9397e6539811d76/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
nsfs /run/docker/netns/5218eef96ce4 nsfs rw 0 0
shm /var/lib/docker/containers/0c21b3ff8c73371bb69040dc9c85f329785693caaafdfa3fd5e7bafe60b81554/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/762a928d2bddfc8ad88b923e0297f9a56dbe279cc879bc2140b53cf1ecc1de85/merged overlay rw,relatime,lowerdir=l/OL76DWXSF25545OFFSRRSB6FOJ:l/QQE7ZKLS6PLMG4VF4L757EF3NN:l/OJEUL2ZSIMIFPSSQTVYIOCOUIF:l/ZLSNUTNJ4MKPMERYKJ42ADR2US:l/NEV6LWAGXXXIIXMKNXQIBNQOGO:l/VDKZL522CRT23UFFVE4YIGSQFK:l/UAP3TDSYT6PZFFMU6HRQHDG35Y:l/BFFL2KKCJIOG2QLF6LUFLRJNX2:l/CXBQONISSYY7XHCXPMG77HF43K:l/D7W5RWDZIQG6LUZJ3RUGJDF6KF:l/V4LTI367CICWWNFIRV7CGPVADU:l/2KSVMELMWS4IAWD3F2HEUP53G3:l/DVNJ6XNSBHOLQCEUXXNAKBANYK:l/RAXHR5BTNXG5AI32RB5KSDVSO3,upperdir=762a928d2bddfc8ad88b923e0297f9a56dbe279cc879bc2140b53cf1ecc1de85/diff,workdir=762a928d2bddfc8ad88b923e0297f9a56dbe279cc879bc2140b53cf1ecc1de85/work 0 0
shm /var/lib/docker/containers/08b14045cf19e26bed0fa8e0a4633849d9b57170d927fa430112ed860ea18914/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
overlay /var/lib/docker/overlay2/c307d14416a15459c7245fb949e9819b7cd2298ae0f086a4e88521f2165a33f0/merged overlay rw,relatime,lowerdir=l/6IMPLUFP7HVJVPSLHLIISO56Z5:l/IB3354XF56K34J4JADLASMQXII:l/44LH56Z3R7QNNWY5KRGTKJNQZI:l/FVMWU5TEDWL34IM6CMEQCREKD2:l/VKIEK4JNKDIPDUZO5WUQLT76ZY:l/TBE5TVZ3SJQ7OTYQW2TXKOQFU3:l/NQGYHZOULW6PURUMFOFK65JDLM:l/42XHXR7GUIHLKCKGALOV2YQ66K:l/7JGH7T6HWHKFLEAYJYV5LG3IOO:l/LUWX7JYEET7ZA6WFHPWMBF2YNC:l/OE7BYYQQAPD2KQ5NIBMH7IO6LL:l/2TLYJDSEY4C7YYQW5HTI4RFFQK:l/ZD3EQBCW3CASDWKLRLIXGYTQ55:l/BADYPTXBCVH6RKYR27EJQBZBVU:l/GDDE23U3ZS5QP7SH7EUCQXKMTS:l/C6MH6I25HMNO5WHZYRJX3HZHNX:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=c307d14416a15459c7245fb949e9819b7cd2298ae0f086a4e88521f2165a33f0/diff,workdir=c307d14416a15459c7245fb949e9819b7cd2298ae0f086a4e88521f2165a33f0/work 0 0
shm /var/lib/docker/containers/f2a3f84b5d5aa10a8726eee4b21f747e988c01b927001f0e53347941933c0455/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=404668k,mode=700 0 0

But on prometheus I can see:

matthiasr commented 7 years ago

When you exec into the container, what is /proc/mounts there? Running node exporter in a container is tricky because Linux doesn't let it see the world as it is.

Try running it in privileged mode, that helps a bit with this, but not completely. Node exporter works best if it runs in the root namespace.

On Thu, Jun 1, 2017, 20:02 Yaroslav Molochko notifications@github.com wrote:

Host operating system: output of uname -a

Linux lhss-nyc2-prod-15 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:54 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter -version

/ # node_exporter --version node_exporter, version 0.14.0 (branch: master, revision: 840ba5dcc71a084a3bc63cb6063003c1f94435a6) build user: root@bb6d0678e7f3 build date: 20170321-12:12:54 go version: go1.7.5

Are you running node_exporter in Docker?

node-exporter: container_name: node-exporter image: "prom/node-exporter:v0.14.0" network_mode: host mem_limit: 60M restart: always command:

  • '-collector.procfs=/host_proc'
  • '-collector.sysfs=/host_sys'
  • '-collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($$|/)"' ports:
  • "9100:9100" environment:
  • CONSUL_EXPORT_NODE-EXPORTER=9100 volumes:
  • /proc:/host_proc:ro
  • /sys:/host_sys:ro
  • /:/rootfs

What did you do that produced an error?

Just check filesystem status from prometheus server, node_exporter reported filesystem as RW, but it is RO for real. /proc/mounts file

cat /proc/mounts sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=2015212k,nr_inodes=503803,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=404668k,mode=755 0 0 /dev/vda1 / ext4 ro,noatime,errors=remount-ro,data=ordered 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,noatime,nodiratime 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 /dev/vda1 /var/lib/docker/overlay2 ext4 ro,noatime,errors=remount-ro,data=ordered 0 0 nsfs /run/docker/netns/default nsfs rw 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/docker/overlay2/3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/merged overlay rw,relatime,lowerdir=l/3BG4CPAQ53BVJ6MONFIOQTI6TP:l/CP3RYUIBQXLJH7YN7D65LZ3KDJ:l/GFWF3CBKLNI34UGY7TEULWC4T6:l/HF23JW4HM6LDKAR7C7VW2WFVRT,upperdir=3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/diff,workdir=3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/work 0 0 overlay /var/lib/docker/overlay2/b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/merged overlay rw,relatime,lowerdir=l/XF2NMAJUM74T7U3MGQRKGBUDM6:l/JIRZCYFTPOODFPMYKMTCTIJ5SB:l/PEVBKEUM3JKO4QS4OE7KX6STKX:l/IBENPFOXDDXO4LJ4IZ3QVSUMXB:l/EYQNXNBLIIJFKIOVYHYU5JSOB3:l/ZB2T7R2ANOK7WHREXPDXKIIXEM:l/F5MQQ27KT43F4QGUIA6HSZFD2Z,upperdir=b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/diff,workdir=b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/work 0 0 shm /var/lib/docker/containers/abdd6c4e1bf9e63bf6afab17913d1b7c434d32df0659163a16e6ca4cbfa8b346/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/merged overlay rw,relatime,lowerdir=l/BRXXP3442WG4QT6G27YBYADKEM:l/N2PZTBZJHWLCTTJWBW4WVU3FCT:l/ATVYPEIKPQFUOBIVS3LWIKGTQK:l/2CL5YUBRWWTRD563QX6KMVER74:l/MIK5KHZOJDC2NO7GSZH2S5MDMK:l/HQQ7D35VIVIEJGWJIIW7FP4N7J:l/YAZAP37C5YZDHGKLXQYDT5Q2VW:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/diff,workdir=686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/work 0 0 shm /var/lib/docker/containers/9e1da532483b0632fccf489931520a08f1833eda2b05d6394e29d507a0977c31/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/merged overlay rw,relatime,lowerdir=l/6RP6XWW6BHYW5V4LRFSYJESODU:l/BFJDVRPVBJOLNQQGPCZH6V3B4B:l/PCAVIKPKGVV2R3QC6LVAUHQL5K:l/Z2L2NKG4ZXYK56XLSHDH4RFRXV:l/PZM57C3GVDXGRQS7PK7K4AVUM6:l/QBPTKHSWVJXL7TWHN5OQ2EHNNP:l/S5367OB7IR5VXRDDGODFLQWKX3:l/267KQMWEPLGOWGC67ZENWRZVG5:l/6YQNUXFRJPY72T7NWW2KNQTOYE,upperdir=31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/diff,workdir=31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/work 0 0 shm /var/lib/docker/containers/c7e7945e69fc410589bd20196b87dd1ffc707de2fe8f6f0c177b71e16f6fb029/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged overlay rw,relatime,lowerdir=l/XWEPIUFAPP7ZI7GICWFLWYR2ZV:l/R5PG7LY6KPIUEYEZIQFPM55HZL:l/2YYFFSQILMJNGDI2EY64BA3DCK:l/M6FZJXQE2J3O4QLIF3Z6OV3IYV:l/OAPL6OCFMDPJSCFK5H5OKUZBB2:l/TVOOCBC2GTB3FRR3MMU6NAKDYF:l/HGVPR35DTAXOOIWH74Y4BSOUCK:l/4LMQBXJDIVIK2D5DI3WW473ZXZ,upperdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/diff,workdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/work 0 0 shm /var/lib/docker/containers/dc101ac35f7edc560434c55d9485c00b832c47bdbfa4035f9af37b41e5dd0133/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/e3dd5f9f858e178dd1791ea0ac75873891ebc03a7143ce5282b3f29ead7b9057/merged overlay rw,relatime,lowerdir=l/YBD2K7GJLC6X4AG6SKIVKYBJZ3:l/PNSDAF54YMYHCTAIFBE7XL7HCX:l/H7WX2VXLDYCRUDUKJN7SCMKXOQ:l/METFBHQH7V4JNNR6UVASJOJXPE:l/CSFTHUBNUH3RC4UBURPF54ZGM2:l/7YSPDYBNLWOQ7CWDYOQLV3YZPD:l/562CEQO75XTY6UXU4TX226T535,upperdir=e3dd5f9f858e178dd1791ea0ac75873891ebc03a7143ce5282b3f29ead7b9057/diff,workdir=e3dd5f9f858e178dd1791ea0ac75873891ebc03a7143ce5282b3f29ead7b9057/work 0 0 shm /var/lib/docker/containers/33f02ada6e6e3698742b0eab52982080ab5589567724580703fe8885b00c16f6/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/6bf604f6676449326e15e01a01d021da4fa150f219d3084b95d7b47ceafc3559/merged overlay rw,relatime,lowerdir=l/DHT2VMJI2JSICXSBMAL2AZCONB:l/BADYPTXBCVH6RKYR27EJQBZBVU:l/GDDE23U3ZS5QP7SH7EUCQXKMTS:l/C6MH6I25HMNO5WHZYRJX3HZHNX:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=6bf604f6676449326e15e01a01d021da4fa150f219d3084b95d7b47ceafc3559/diff,workdir=6bf604f6676449326e15e01a01d021da4fa150f219d3084b95d7b47ceafc3559/work 0 0 overlay /var/lib/docker/overlay2/65cf9af585b0f3903429de31488b31f147a14c13a2d12fb11823ed22529cf1a7/merged overlay rw,relatime,lowerdir=l/AEXD42UDHUPIYKSJ332RSVBM6E:l/QFSLFRRXC5NNYDZ2DAJZJFZZ4G:l/BZ7B3JSSLOV7HD54BCTGCRWM2H:l/52NOYFLRTBSHVFZ5NVYATGCY73:l/UIEAFNER25HREXE6OBFWS3UP7B:l/M6MZIHXRZPCJT6M74YXABF3YYA:l/W64RVD2PBLFLTXGBIVXFSV57OS:l/SUSMWOMW5CRCFPV2UBBO2ZOFXL:l/JO4LDPRR4TVFSECKZ7SE5WH5UL:l/MSDV4B2OC3MCUKQM3XP5BR7CIS:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=65cf9af585b0f3903429de31488b31f147a14c13a2d12fb11823ed22529cf1a7/diff,workdir=65cf9af585b0f3903429de31488b31f147a14c13a2d12fb11823ed22529cf1a7/work 0 0 shm /var/lib/docker/containers/57f22cdce9f60c69726e21b29a7ddd4a94e2b03f8d0e58754e879ecb9fe6ca5c/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /var/lib/docker/containers/51b68b7663a0a75ebe96731ca92e7d1ca23bed087021424b0d41c368973a8c6f/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/76c8de59cdc2 nsfs rw 0 0 overlay /var/lib/docker/overlay2/4d0741cf64a57fe38874358ec2e56ec77301f60b74046c5c3b9f5a4140d22b26/merged overlay rw,relatime,lowerdir=l/E26DOIDEWL4JTIVBC5J5YLNAQN:l/VH6ZYNLP6LYYDZ3LJZA7UT3JCY:l/ICPBU5ZTIDXL6FK3I6MN6F5C76:l/WQWT4DPTG3SEHUBNVMLIVI6I3X:l/562CEQO75XTY6UXU4TX226T535,upperdir=4d0741cf64a57fe38874358ec2e56ec77301f60b74046c5c3b9f5a4140d22b26/diff,workdir=4d0741cf64a57fe38874358ec2e56ec77301f60b74046c5c3b9f5a4140d22b26/work 0 0 shm /var/lib/docker/containers/bc74830ec1b4391acf7b7a9ed353366489b629922b6a744ea286a944d0704b02/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/2c65c0a47b5bcf34d8cb21aab01f573e91c7488201a66cbf68c1d7c296c0fa9a/merged overlay rw,relatime,lowerdir=l/7YS3Y5ISIPZH6AEUZQW36Z7C7V:l/WW42JWB6GTBY6GRWMB4QXJYDTY:l/AFG2P5SSBKAIWOIG2AXRNOG75O:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=2c65c0a47b5bcf34d8cb21aab01f573e91c7488201a66cbf68c1d7c296c0fa9a/diff,workdir=2c65c0a47b5bcf34d8cb21aab01f573e91c7488201a66cbf68c1d7c296c0fa9a/work 0 0 shm /var/lib/docker/containers/56d8d2b8292b150ab9d322d50875411547acb7b4076e264a03be11613eccf305/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/f139d78b7661ee11e567c520bd6d51862ffc36851fa4d29f4fe168d0814c542d/merged overlay rw,relatime,lowerdir=l/BE57KFYMA7AKIK3T5C2LRSHY32:l/N5G4SVHKQHRYCUZVHDU3DQKLZT:l/GFWMA4VHY5TBHADKL56B2RE3ED:l/HRDONYOUN37K3MH5CD662F4NXM:l/JAMKPS2YLPWRIN3HOVM36XERV7:l/CF4EK4MKJM3PCS3AS6F6YIME7V:l/DMWX4TQ2HNJAODZTCPU65XDIZ4:l/BDF6LYJR2K3ORC7KH4S5E54JXB:l/GSK45JTGLMDDZSHCBNLWLKWL4P:l/S5HZMUW7QU4G6MHCNMX7R3JAQZ:l/3XCRIB6HY6OQTHWBCBUZGZDS2G:l/2PUJTWPRMQ5VOXXCNSONR4F2SS:l/GOK6L2AE2I6WIHCT3W4T4DLTPM:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=f139d78b7661ee11e567c520bd6d51862ffc36851fa4d29f4fe168d0814c542d/diff,workdir=f139d78b7661ee11e567c520bd6d51862ffc36851fa4d29f4fe168d0814c542d/work 0 0 overlay /var/lib/docker/overlay2/04fee163764e681b89820ff1d298771d3a5739d7dbd0cb386b63752e5aea531c/merged overlay rw,relatime,lowerdir=l/6VLFNU7EQE7GEVGQZHR4IMLRCS:l/P3P2PPZSTM4Z3EERAZPFY5K2SA:l/FZ2VJOZ4NJF5G6WVOXK4M452DM:l/LIBTIUPHBYQ5SRRWLW7XQ4SYXZ:l/MKEPOLZRJULHJNTBIGRHJB3NR7:l/3DNQ5SVC2T252UUCS664WXAQ7M:l/IPT32VF3OVOUXST3EEAXRWVOZP:l/MSPMOOEAWGE2TSCLY2BXT5JO3Y:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=04fee163764e681b89820ff1d298771d3a5739d7dbd0cb386b63752e5aea531c/diff,workdir=04fee163764e681b89820ff1d298771d3a5739d7dbd0cb386b63752e5aea531c/work 0 0 shm /var/lib/docker/containers/6b44bfa202ffce2fce5d6bc1981727269b5dd07526baa5410ebcf2f452c9d73d/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/8ccf77d59af5df50e0743ac9378b003dec25fbc03cba1d77c017c2f69d5c9939/merged overlay rw,relatime,lowerdir=l/2YGDZFBDJDPUD6HJFU57E725G2:l/IBDR27CWL2NL53BM6U6SEUXYIP:l/I6VCBQWMBELZZBYB7OEKQRPUW3:l/K7TLIJJOFUTGTHLN2CBQGBOCDB:l/6RQJWH25GMDFSKVJ7IKXDW54WR:l/PFRHQUFA4GEXASUAU2CR6VRK57:l/MA4C7V5OGKBSJ5KAU4TIFPFEET,upperdir=8ccf77d59af5df50e0743ac9378b003dec25fbc03cba1d77c017c2f69d5c9939/diff,workdir=8ccf77d59af5df50e0743ac9378b003dec25fbc03cba1d77c017c2f69d5c9939/work 0 0 overlay /var/lib/docker/overlay2/77b20d8845ae32ccc609ed80523ec9f4eff6f9f8bcd821c0e9edebdc00bf0f78/merged overlay rw,relatime,lowerdir=l/K3C4QQ6SXKCOCCU6CCDAKXXADB:l/CFMCKHVU24W3ZQLFNHJL62MAFD:l/RLF5XBGRYE6W6OWTJOUDRGPWOM:l/DBK2UXZCY73SKN27GT3XNCAAJD:l/5C2FDNSYQDRTU6AZH3RDIFNXCF:l/JORA2F6B2HXS7XHBZ6VSWZ2ISU:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=77b20d8845ae32ccc609ed80523ec9f4eff6f9f8bcd821c0e9edebdc00bf0f78/diff,workdir=77b20d8845ae32ccc609ed80523ec9f4eff6f9f8bcd821c0e9edebdc00bf0f78/work 0 0 shm /var/lib/docker/containers/dc4ed9e21112485d6ee67ca1f5b64c16732c1581420c1c0b9d5b312f475d727a/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/7e68f15a2311c6b4421b0053e376e9cb1d3e87ee62b5991fb7ad5d6c45477b09/merged overlay rw,relatime,lowerdir=l/7UWLP6IX7NJ4YQB72AYPKJHZRK:l/5OUOAAD2FZAAS2AQUPSW5OVHAC:l/X6YI4PBOGFOJZDPYYCKKKGBOEA:l/XTTQTESPNTYH7TMEQEXCDVHNAM:l/YXUJVF3ZXTUL4YVDC65XNF5HZA:l/2UYGOOL55QR66R6WIXWDB6WY3P:l/KJ4GQPZOPZ6RLEBQPMHHNXDOTF:l/ZAER47HQ5ZF3SRGHJ45GMDHLHF:l/WTDIYWY2DEGN4D4YZZ53LGE2BU:l/RZ3E23FRTIBMEDROXSNNNB57TO:l/V2W5AI2CTZJPOFNL37R6UOQPCE:l/S3QAIREI2HSXGJFHOAP6DCNKLN:l/PCGNUWUWO25NVC6DBBEB4QLNTU:l/OGSKYF446IHCN44EBCNRQX6HRX,upperdir=7e68f15a2311c6b4421b0053e376e9cb1d3e87ee62b5991fb7ad5d6c45477b09/diff,workdir=7e68f15a2311c6b4421b0053e376e9cb1d3e87ee62b5991fb7ad5d6c45477b09/work 0 0 shm /var/lib/docker/containers/cd50af9091b8280c4419f339a7fbcd1542c2958ec93a915f1797943c8275d80a/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/4662af0b733601ce15e9d68692e50b2f492987a29b7ec3a676486e10865da064/merged overlay rw,relatime,lowerdir=l/VYUPKYB4PQHKAPYSSD6AV3IQPS:l/5OUOAAD2FZAAS2AQUPSW5OVHAC:l/X6YI4PBOGFOJZDPYYCKKKGBOEA:l/XTTQTESPNTYH7TMEQEXCDVHNAM:l/YXUJVF3ZXTUL4YVDC65XNF5HZA:l/2UYGOOL55QR66R6WIXWDB6WY3P:l/KJ4GQPZOPZ6RLEBQPMHHNXDOTF:l/ZAER47HQ5ZF3SRGHJ45GMDHLHF:l/WTDIYWY2DEGN4D4YZZ53LGE2BU:l/RZ3E23FRTIBMEDROXSNNNB57TO:l/V2W5AI2CTZJPOFNL37R6UOQPCE:l/S3QAIREI2HSXGJFHOAP6DCNKLN:l/PCGNUWUWO25NVC6DBBEB4QLNTU:l/OGSKYF446IHCN44EBCNRQX6HRX,upperdir=4662af0b733601ce15e9d68692e50b2f492987a29b7ec3a676486e10865da064/diff,workdir=4662af0b733601ce15e9d68692e50b2f492987a29b7ec3a676486e10865da064/work 0 0 shm /var/lib/docker/containers/d396e24736abdc0ae329e3308a2b3165f30a0bb531fec13aeb1e7bb796c1d529/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/1099a380b47598260d4005613e6333175d759b11f1a6b04fc48b2a31104a28cc/merged overlay rw,relatime,lowerdir=l/673A437C76AY3YW3WSMZXIJEBM:l/SWMKAAU2KKRZY6HSJCJ6QUO6M6:l/755IX5QKEAJOOIIU6NT43UY26D:l/NOEJE7GJIAYBFVXNQPEPBENEPG:l/H2P6RE5T4YI5SEXSMP34T2F7FB:l/562CEQO75XTY6UXU4TX226T535,upperdir=1099a380b47598260d4005613e6333175d759b11f1a6b04fc48b2a31104a28cc/diff,workdir=1099a380b47598260d4005613e6333175d759b11f1a6b04fc48b2a31104a28cc/work 0 0 shm /var/lib/docker/containers/af6beeccb90fa85eb516c96438cd2f49e82eb7a4a92ae47411d885b29655d847/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/5f6bda2aa599 nsfs rw 0 0 overlay /var/lib/docker/overlay2/5174be1b43da19e5da75a9a3679fa99cf393e89372c484cc845a91e0c6489d61/merged overlay rw,relatime,lowerdir=l/6ABO6S6RSNOO7HR7CEY55XGASR:l/63DUYF2VS4IKNYQ54LNF2KHJBY:l/6YLX6HSZ6FCQL7GWVPD2WLQCWY:l/6DKENABXIBZC25DZVZC5F4F6N4:l/LHVECFQKV6ZXASUAZPNSHHETHB,upperdir=5174be1b43da19e5da75a9a3679fa99cf393e89372c484cc845a91e0c6489d61/diff,workdir=5174be1b43da19e5da75a9a3679fa99cf393e89372c484cc845a91e0c6489d61/work 0 0 overlay /var/lib/docker/overlay2/29f186fd8dea22da96b841c9335d6f6c7359a6b525dcfc0cb9f730103a464d90/merged overlay rw,relatime,lowerdir=l/MG2H2WJTU6DNBZUVOJZH5MLTGQ:l/P5BU3TFKZGQMSGIHTLHBCNOY5G:l/YYAKJSWIVFTGAJZA2YQ43QNIBR:l/IUVVROLNZNUVX5PYBUBTQ3PNZT:l/TZQ655KGNSCJ5GATH676U3PVP2:l/JHNROZ5V5WMFLC43CTKCL55QEY:l/MA4C7V5OGKBSJ5KAU4TIFPFEET,upperdir=29f186fd8dea22da96b841c9335d6f6c7359a6b525dcfc0cb9f730103a464d90/diff,workdir=29f186fd8dea22da96b841c9335d6f6c7359a6b525dcfc0cb9f730103a464d90/work 0 0 shm /var/lib/docker/containers/a930efae2d1052337ef38fb281291f057626774ed9592355b824c481722b5e30/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/1f0be0883304cfa77754501f6a6f3a6385457ed018dd0bcf41e8461a401e1a5b/merged overlay rw,relatime,lowerdir=l/LY42C4Y4VGNTIGVS7PLZGCIW4A:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=1f0be0883304cfa77754501f6a6f3a6385457ed018dd0bcf41e8461a401e1a5b/diff,workdir=1f0be0883304cfa77754501f6a6f3a6385457ed018dd0bcf41e8461a401e1a5b/work 0 0 nsfs /run/docker/netns/afe124c50f0d nsfs rw 0 0 shm /var/lib/docker/containers/107e222bdca4262c7693545d6ae5d726eb6d79655fe4137ea0f9d6dfb477c418/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/29d06c0e2b5175420bfd217fda80445eb0184a191ac534015b510f87b6f8946e/merged overlay rw,relatime,lowerdir=l/U6FKGZDAFB4AEK76NGGRD4HL3X:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=29d06c0e2b5175420bfd217fda80445eb0184a191ac534015b510f87b6f8946e/diff,workdir=29d06c0e2b5175420bfd217fda80445eb0184a191ac534015b510f87b6f8946e/work 0 0 overlay /var/lib/docker/overlay2/40b152b91022c658f412c9f972667b4e2857836a64af7ac9437cb8ceb2f53491/merged overlay rw,relatime,lowerdir=l/GHZOCRA76KI66UESTMRDOG5BMW:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=40b152b91022c658f412c9f972667b4e2857836a64af7ac9437cb8ceb2f53491/diff,workdir=40b152b91022c658f412c9f972667b4e2857836a64af7ac9437cb8ceb2f53491/work 0 0 shm /var/lib/docker/containers/e4123df790b1461db62dbdb72fbe5612274e62c5922e6072053e5e2fba92421d/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /var/lib/docker/containers/5d9a8a4625f3c70d0761a5bb74bd8632333f689394038b278af525608fbabe70/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/172e701a8234 nsfs rw 0 0 overlay /var/lib/docker/overlay2/12002438dbb35d69e53faeeee0a8afaf25734d150fbfa05f5eea2f25ac9f02ce/merged overlay rw,relatime,lowerdir=l/LZDWILO3K226HFFC4UKIGH3ETE:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=12002438dbb35d69e53faeeee0a8afaf25734d150fbfa05f5eea2f25ac9f02ce/diff,workdir=12002438dbb35d69e53faeeee0a8afaf25734d150fbfa05f5eea2f25ac9f02ce/work 0 0 shm /var/lib/docker/containers/4b820a0a8bbee46507800b8b9d72d14d93c55037aeae5e53cd9084e2a761577b/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/648777e6cbc1 nsfs rw 0 0 overlay /var/lib/docker/overlay2/88210d1097537c404ee15b6443fe84feca58f570fb6a6461dea7def1aab6cfa0/merged overlay rw,relatime,lowerdir=l/X67DZKT3SPH6GIKBW25OA7AIJ2:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=88210d1097537c404ee15b6443fe84feca58f570fb6a6461dea7def1aab6cfa0/diff,workdir=88210d1097537c404ee15b6443fe84feca58f570fb6a6461dea7def1aab6cfa0/work 0 0 nsfs /run/docker/netns/51eb96785424 nsfs rw 0 0 shm /var/lib/docker/containers/5d375a99310bc125aee862f4ff154c13052c7d7f7ddd4d5a4a4ff49e4296261a/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/7072177a27b5 nsfs rw 0 0 nsfs /run/docker/netns/4e505f39baa3 nsfs rw 0 0 overlay /var/lib/docker/overlay2/279a9f48307dceec2985e66009ca08cf0a859e2a3cdc6a105ff22ece46cb6743/merged overlay rw,relatime,lowerdir=l/ASGWHWGYCJDIDH5WCLHYRVOB3U:l/SSBYLJRQCOQRM6G3Z6AZRPXDIY:l/Z4XANGU5UBPAE2BN6XRLDWEVJM:l/6JFFALMYSXPD3EPKW3JDFJY5KE:l/JTA2PPOT6Y6EXFBWZRLW3DG2ZF:l/MWWMO6PQMRMZOYTYBQKGDL5TU5:l/6WSQB2TPTZKHYZHPWO3GZWIXCD:l/MGDALBCITEZTR5QYSLEM45CFQJ:l/57NSTC2PO3HVX7ZBNL2HHTMYQA:l/WJRGPI7TJK5XPSIS4OWNSFNFRD:l/NYIKCPYM2FG4QW7A4ZLPYXXAGU:l/6FDNL6NCIFPVKCEE2KOD2MFQ5F:l/GXWTDPHDNPRRNXRB5OKK2JML7R:l/NAQEOYNTO2WORLCGB2Z4LTI4BT:l/IXMATN5H64UHH26DXGAUJBRS3H,upperdir=279a9f48307dceec2985e66009ca08cf0a859e2a3cdc6a105ff22ece46cb6743/diff,workdir=279a9f48307dceec2985e66009ca08cf0a859e2a3cdc6a105ff22ece46cb6743/work 0 0 shm /var/lib/docker/containers/627a298efe64cf73d4aba3685ebd2f217b59c6309f1a03416715ba5464f4f272/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/9cf4b56137f8 nsfs rw 0 0 overlay /var/lib/docker/overlay2/956b3bc1da9859fc131264accbdb10f06b7a1d1495b048a8d753b15ebbb697ff/merged overlay rw,relatime,lowerdir=l/AVEK7TFPLJSI7HXGS2P3UYZRBY:l/HLANIVF7ZYWRWYGDDLVU6VZBR5:l/G6IB4BYW6NQ54FIDJ7U4QVAOYN:l/S34BJ25WX4AB5HRBT2JQJL5CZY:l/T2EBQ6PUKNRNQGN3OKJEKJI2L7:l/OCHL7WYND3NVJAGBF4PHOKMB6U:l/PRA2HVVNG72PMXSTDZDCPGMYQF:l/QWIXKB2AI6GY64MHEVD7FW2ETD:l/A2POFG2F4JUSU4UJ44TB4NHDAL:l/SIZ5JFD2VZNGIED2LQALDALDAR:l/I7YX3US37NXNIVOBM6GL7ZXVRQ:l/GPHJ357FFESHMQVXWQRA2YARQG:l/MICHTM542NQSUSONQX7DR3OANI:l/XDL4GVT2H2F2HW77RZRNMV5SSF:l/VD7ORKW6N5HGNYK662NM62UGUZ:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=956b3bc1da9859fc131264accbdb10f06b7a1d1495b048a8d753b15ebbb697ff/diff,workdir=956b3bc1da9859fc131264accbdb10f06b7a1d1495b048a8d753b15ebbb697ff/work 0 0 shm /var/lib/docker/containers/dfff0fbe06984a5dfd6c331ab0dd5670aceda7bb68609b3bb61f836139b0d932/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/50fc677bce4b127adce650bf19f1aa18b0d9b887a76147fd9dd0b5aebb7f77a7/merged overlay rw,relatime,lowerdir=l/VCBIMGEMIODEP6JP4ON5WUPXRR:l/U632HQR3GJWKYEYTRSHYHEHSHI:l/R7BYP7XOQDRM7M3O5HXXYLCL3F:l/OUQMUGIOQDZFS5JXCFPI3QPNRJ:l/52BPGQI3K2HTP652TV5QS7XOKN:l/IIK6ADFKT2BEFDECUMDZ2IRPDB:l/UJWVQ2CDVLVNY5TP6RNCNNLS7A:l/UPY55VKPKLPO4XJMDQ7OUMDRT7:l/YWW3MC7S5EISNUCYTRXNCSXM6T:l/EO6BYWDSTO2R4LNQLRCCHLCK7N:l/WIGHSYPZQX3SAI6L2FVLZL4R3B:l/BWPTKHH2DCPQSW6KVGJO52AGVK:l/OJTLREGVSNY2PMP3UEY6UOXLA5:l/G3TLSZX3OO2NAJIOWRCY6CAVXH,upperdir=50fc677bce4b127adce650bf19f1aa18b0d9b887a76147fd9dd0b5aebb7f77a7/diff,workdir=50fc677bce4b127adce650bf19f1aa18b0d9b887a76147fd9dd0b5aebb7f77a7/work 0 0 shm /var/lib/docker/containers/a6bff365eb62526b9bddf45888e88c634724831c7b3f0a36ad40deabecfff659/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/f90fd68a05a2 nsfs rw 0 0 nsfs /run/docker/netns/45741e5fc002 nsfs rw 0 0 overlay /var/lib/docker/overlay2/e90727dcdf49455c8e73050690353a9661ccda258507179423a2fd5d440dd97c/merged overlay rw,relatime,lowerdir=l/TD6MO4LPIHXW7S55HV5ASBIGUF:l/Z74INQS3QUKS33BH5SWFKYECAL:l/QWYJ33TPVNMM3G43GEYMNVP6WP:l/JUCWLPSGVW27SSNZRJB53SFZUE,upperdir=e90727dcdf49455c8e73050690353a9661ccda258507179423a2fd5d440dd97c/diff,workdir=e90727dcdf49455c8e73050690353a9661ccda258507179423a2fd5d440dd97c/work 0 0 overlay /var/lib/docker/overlay2/a8e76f39b139bb076fc98033a4c4cbace5ec17c53f2c55e78c32db23f72e4b01/merged overlay rw,relatime,lowerdir=l/Z53CAPAOA5W5XTPWQWUQ77WRXO:l/QNAR5EZDEO34TC7VH7BGOU6XJM:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=a8e76f39b139bb076fc98033a4c4cbace5ec17c53f2c55e78c32db23f72e4b01/diff,workdir=a8e76f39b139bb076fc98033a4c4cbace5ec17c53f2c55e78c32db23f72e4b01/work 0 0 shm /var/lib/docker/containers/fe57216cd6b7716094531d33a2bf6e78b58d7bb04431ca75fbcb437cac86553b/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /var/lib/docker/containers/8fb3b3a02af0513c3a782e133b9f3f0bd150d9a888c55b75c200978c741b703e/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/81d64a728479fb2fbdf694ee3b03d73e9182236d5bf682a247ad73580a39b08f/merged overlay rw,relatime,lowerdir=l/MJNF24CSO34SH5UVCWZAKAPOCZ:l/RJCFDHYQJ3VOU3EUV24R6K2KR7:l/GUSD7J7VXXSSA4NWYB6NB7JEMU:l/DJHNAAJXCFD6LXATCMO7ZWXJCT:l/XOC6OYPNXLOXWGA3KKV3AMEA72:l/PBMUXCWHVZXUJDX6VMDG6CVT45:l/WU53ZTW7IE4C5GO6O2VBMPGCV3:l/PTAGFOZNVDCAV5EJRTLG3YUXBH:l/3LG6WXCLMOHSKIUZUALCG7CF5Z:l/HSU3GYPQTY442LQPVQGNQ65WXX:l/HV4GPUA5HBWVRKDLEUFTJMRE5U:l/RRYIYPAYVEKT2WCF2SJL3Q3RN7:l/AIEF3JN342NCNOKS2ZEMOZARP4:l/MA4C7V5OGKBSJ5KAU4TIFPFEET,upperdir=81d64a728479fb2fbdf694ee3b03d73e9182236d5bf682a247ad73580a39b08f/diff,workdir=81d64a728479fb2fbdf694ee3b03d73e9182236d5bf682a247ad73580a39b08f/work 0 0 shm /var/lib/docker/containers/fee24d58563a7c0a7afb06cab6488e50d93061dee4d1b2248422c61b76216113/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/653cf6c5e5ce5d2b7eeccb075a9f57bebf974306a91592fb974878f48b38e4a2/merged overlay rw,relatime,lowerdir=l/7RKQVSWIVA4RJGJTDNZS3R7IRH:l/KDNIKWLCPVZT4UMXE2WLA374MQ:l/VT34J2GBQ7LJGZ2IN5DVVIBPYM:l/3LN2PBUYF4F35NKM36IH2E7Y2R:l/AUUYD2YIKWKOILH4X4MNSC5QPE:l/OUMMKWR6IVYYELFDAVD6ZNDVE3:l/WLHKWWMNTFYYQXIANQ5HQLVPDU:l/YGSK7FEL6OZSVAV77EDL6Q2WPG:l/A37U373A235DJLZAN4JPDWCRXD:l/RNQ7SL6ADKOFBIKF66QYB62YTK:l/2T6B32DW5M6TUAHKRHNWYH42MY:l/BGGVNYKWWZE7OF3YVSAAM2V33L:l/CO3MPV4YDA6LCIGA4O4H6T3LAZ:l/FT7PNF3YEAJR7Z5KHJ4S7HOXUF,upperdir=653cf6c5e5ce5d2b7eeccb075a9f57bebf974306a91592fb974878f48b38e4a2/diff,workdir=653cf6c5e5ce5d2b7eeccb075a9f57bebf974306a91592fb974878f48b38e4a2/work 0 0 shm /var/lib/docker/containers/75a22b1b2a0a902b1b6cd59a0c76d7066be30ba9933e434207fdc9439913dc72/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/3d66e96273b9f51ce722743c2b2ce2853e0999fcefd54244d3d6897af13fcd2e/merged overlay rw,relatime,lowerdir=l/PFU36Z4NEVIB355LBVMMLX63L3:l/HBTQLLLHMFWRMMFPABPCBWDLNT:l/J4XQW54NS4OTKB7ZZHTI6O2P6B:l/TD33P3JEZUOIR2F2XBMLEPHBVZ:l/AA4DUZUREEHL2SQB3NPUIEEAVJ:l/SDN7PQWNQTYUZWPANFQV5RGFAZ:l/UBIE2GBEVPICZNMJD5HWRESHWW:l/HFOZZ3M3B5TECZEHYZJ4ZSQJ4E:l/HU7XQNH4ZL6NBTHSVVUTDB4HCX:l/67ZPF2ICP2VKW6IPJKSCBVKMMV:l/5YXZSCZROXZ2HKFXRN43TXU6RV:l/FL6LD7IIQF2KDQVG535GWTXRTG:l/XWGFJN4XFHC2XKDHW6DA77D44C:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=3d66e96273b9f51ce722743c2b2ce2853e0999fcefd54244d3d6897af13fcd2e/diff,workdir=3d66e96273b9f51ce722743c2b2ce2853e0999fcefd54244d3d6897af13fcd2e/work 0 0 shm /var/lib/docker/containers/bcf3097df526ff65237f29a50ebdf4528ee2727fec1ecc4cefd36e8d16c97e02/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/0927d561f4b49ab231f4e0b23fba2cb5810022df0bca01541e5b5fb5db56958a/merged overlay rw,relatime,lowerdir=l/OA5VYUBHW4ON3TEOEV3NGPST6P:l/PTCUOG5TPCGBDRWLNAIZSIBNAP:l/KZVZKNTLAQHJXYEDZIZRJU6AS6:l/G77BPTXLPHEBWKWMGE327PT2P5:l/DN4IEJ3WSIEUMGGRNHZ2B2LVAC:l/AXGCS4BJWFNPY3LEU644XXG2FP:l/A6AC3F4JO5XZ64RYJMR5AYOYUC:l/BIMN7U245V6ZJBGKAIJIPRHZC7,upperdir=0927d561f4b49ab231f4e0b23fba2cb5810022df0bca01541e5b5fb5db56958a/diff,workdir=0927d561f4b49ab231f4e0b23fba2cb5810022df0bca01541e5b5fb5db56958a/work 0 0 shm /var/lib/docker/containers/7b1909e0a9a3d63669d61dd65c92b8556613e52255b92241d9397e6539811d76/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/5218eef96ce4 nsfs rw 0 0 shm /var/lib/docker/containers/0c21b3ff8c73371bb69040dc9c85f329785693caaafdfa3fd5e7bafe60b81554/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/762a928d2bddfc8ad88b923e0297f9a56dbe279cc879bc2140b53cf1ecc1de85/merged overlay rw,relatime,lowerdir=l/OL76DWXSF25545OFFSRRSB6FOJ:l/QQE7ZKLS6PLMG4VF4L757EF3NN:l/OJEUL2ZSIMIFPSSQTVYIOCOUIF:l/ZLSNUTNJ4MKPMERYKJ42ADR2US:l/NEV6LWAGXXXIIXMKNXQIBNQOGO:l/VDKZL522CRT23UFFVE4YIGSQFK:l/UAP3TDSYT6PZFFMU6HRQHDG35Y:l/BFFL2KKCJIOG2QLF6LUFLRJNX2:l/CXBQONISSYY7XHCXPMG77HF43K:l/D7W5RWDZIQG6LUZJ3RUGJDF6KF:l/V4LTI367CICWWNFIRV7CGPVADU:l/2KSVMELMWS4IAWD3F2HEUP53G3:l/DVNJ6XNSBHOLQCEUXXNAKBANYK:l/RAXHR5BTNXG5AI32RB5KSDVSO3,upperdir=762a928d2bddfc8ad88b923e0297f9a56dbe279cc879bc2140b53cf1ecc1de85/diff,workdir=762a928d2bddfc8ad88b923e0297f9a56dbe279cc879bc2140b53cf1ecc1de85/work 0 0 shm /var/lib/docker/containers/08b14045cf19e26bed0fa8e0a4633849d9b57170d927fa430112ed860ea18914/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 overlay /var/lib/docker/overlay2/c307d14416a15459c7245fb949e9819b7cd2298ae0f086a4e88521f2165a33f0/merged overlay rw,relatime,lowerdir=l/6IMPLUFP7HVJVPSLHLIISO56Z5:l/IB3354XF56K34J4JADLASMQXII:l/44LH56Z3R7QNNWY5KRGTKJNQZI:l/FVMWU5TEDWL34IM6CMEQCREKD2:l/VKIEK4JNKDIPDUZO5WUQLT76ZY:l/TBE5TVZ3SJQ7OTYQW2TXKOQFU3:l/NQGYHZOULW6PURUMFOFK65JDLM:l/42XHXR7GUIHLKCKGALOV2YQ66K:l/7JGH7T6HWHKFLEAYJYV5LG3IOO:l/LUWX7JYEET7ZA6WFHPWMBF2YNC:l/OE7BYYQQAPD2KQ5NIBMH7IO6LL:l/2TLYJDSEY4C7YYQW5HTI4RFFQK:l/ZD3EQBCW3CASDWKLRLIXGYTQ55:l/BADYPTXBCVH6RKYR27EJQBZBVU:l/GDDE23U3ZS5QP7SH7EUCQXKMTS:l/C6MH6I25HMNO5WHZYRJX3HZHNX:l/G6YBVJXRNBHZJFD3VI7BF4I2V7,upperdir=c307d14416a15459c7245fb949e9819b7cd2298ae0f086a4e88521f2165a33f0/diff,workdir=c307d14416a15459c7245fb949e9819b7cd2298ae0f086a4e88521f2165a33f0/work 0 0 shm /var/lib/docker/containers/f2a3f84b5d5aa10a8726eee4b21f747e988c01b927001f0e53347941933c0455/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=404668k,mode=700 0 0

But on prometheus I can see:

-

request

node_filesystem_readonly{host_name="lhss-nyc2-prod-15",mountpoint="/rootfs"}

response

node_filesystem_readonly{colo="nyc2",container_name="node-exporter",device="/dev/vda1",fstype="ext4",host_name="lhss-nyc2-prod-15",host_type="lhss-nyc2-prod",instance="159.203.108.219:9100",job="consul-services",mountpoint="/rootfs"} 0

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/prometheus/node_exporter/issues/591, or mute the thread https://github.com/notifications/unsubscribe-auth/AAICBqgoPtpxmOLZxKSNcdxmLybD92yeks5r_vykgaJpZM4NtVa0 .

onorua commented 7 years ago

exec output

root@lhss-nyc2-prod-15:~# docker exec -i -t node-exporter cat /proc/mounts
overlay / overlay rw,relatime,lowerdir=l/XWEPIUFAPP7ZI7GICWFLWYR2ZV:l/R5PG7LY6KPIUEYEZIQFPM55HZL:l/2YYFFSQILMJNGDI2EY64BA3DCK:l/M6FZJXQE2J3O4QLIF3Z6OV3IYV:l/OAPL6OCFMDPJSCFK5H5OKUZBB2:l/TVOOCBC2GTB3FRR3MMU6NAKDYF:l/HGVPR35DTAXOOIWH74Y4BSOUCK:l/4LMQBXJDIVIK2D5DI3WW473ZXZ,upperdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/diff,workdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/work 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
proc /host_proc proc ro,relatime 0 0
systemd-1 /host_proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=0,timeout=0,minproto=5,maxproto=5,direct 0 0
binfmt_misc /host_proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sysfs /host_sys sysfs ro,relatime 0 0
securityfs /host_sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /host_sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /host_sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /host_sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /host_sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /host_sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /host_sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /host_sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /host_sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /host_sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /host_sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /host_sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /host_sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
pstore /host_sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
debugfs /host_sys/kernel/debug debugfs rw,relatime 0 0
fusectl /host_sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/vda1 /rootfs ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
udev /rootfs/dev devtmpfs rw,nosuid,relatime,size=2015212k,nr_inodes=503803,mode=755 0 0
devpts /rootfs/dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /rootfs/dev/shm tmpfs rw,noatime,nodiratime 0 0
hugetlbfs /rootfs/dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /rootfs/dev/mqueue mqueue rw,relatime 0 0
tmpfs /rootfs/run tmpfs rw,nosuid,noexec,relatime,size=404668k,mode=755 0 0
tmpfs /rootfs/run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
nsfs /rootfs/run/docker/netns/default nsfs rw 0 0
sysfs /rootfs/sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /rootfs/sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /rootfs/sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /rootfs/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /rootfs/sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /rootfs/sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /rootfs/sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /rootfs/sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /rootfs/sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /rootfs/sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /rootfs/sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /rootfs/sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
pstore /rootfs/sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
debugfs /rootfs/sys/kernel/debug debugfs rw,relatime 0 0
fusectl /rootfs/sys/fs/fuse/connections fusectl rw,relatime 0 0
proc /rootfs/proc proc rw,nosuid,nodev,noexec,relatime 0 0
systemd-1 /rootfs/proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=0,timeout=0,minproto=5,maxproto=5,direct 0 0
binfmt_misc /rootfs/proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
lxcfs /rootfs/var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
/dev/vda1 /rootfs/var/lib/docker/overlay2 ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
overlay /rootfs/var/lib/docker/overlay2/3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/merged overlay rw,relatime,lowerdir=l/3BG4CPAQ53BVJ6MONFIOQTI6TP:l/CP3RYUIBQXLJH7YN7D65LZ3KDJ:l/GFWF3CBKLNI34UGY7TEULWC4T6:l/HF23JW4HM6LDKAR7C7VW2WFVRT,upperdir=3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/diff,workdir=3348b68a022f2adf867ebc07228509d778ff30013b5cb1d4b4782def492900d2-init/work 0 0
overlay /rootfs/var/lib/docker/overlay2/b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/merged overlay rw,relatime,lowerdir=l/XF2NMAJUM74T7U3MGQRKGBUDM6:l/JIRZCYFTPOODFPMYKMTCTIJ5SB:l/PEVBKEUM3JKO4QS4OE7KX6STKX:l/IBENPFOXDDXO4LJ4IZ3QVSUMXB:l/EYQNXNBLIIJFKIOVYHYU5JSOB3:l/ZB2T7R2ANOK7WHREXPDXKIIXEM:l/F5MQQ27KT43F4QGUIA6HSZFD2Z,upperdir=b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/diff,workdir=b32ac2f7593cda50988cd7045b570026be7014a17be7b97259ca9163aa4746c4/work 0 0
overlay /rootfs/var/lib/docker/overlay2/686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/merged overlay rw,relatime,lowerdir=l/BRXXP3442WG4QT6G27YBYADKEM:l/N2PZTBZJHWLCTTJWBW4WVU3FCT:l/ATVYPEIKPQFUOBIVS3LWIKGTQK:l/2CL5YUBRWWTRD563QX6KMVER74:l/MIK5KHZOJDC2NO7GSZH2S5MDMK:l/HQQ7D35VIVIEJGWJIIW7FP4N7J:l/YAZAP37C5YZDHGKLXQYDT5Q2VW:l/MNTFDRGDWKEOYS65DJBUXADZVS:l/62XJIFR2YD4O4NMQDM2VXNOJA7:l/3UEDLD6ZS5YB4ZFHYRQB5PQTXU:l/HDU6HC7A65HGQ4QRFRNXVCKOIE,upperdir=686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/diff,workdir=686ef22f2f3313926025b3cfcfa90a492af746d3129d04c348e1dcecc2cc74f5/work 0 0
overlay /rootfs/var/lib/docker/overlay2/31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/merged overlay rw,relatime,lowerdir=l/6RP6XWW6BHYW5V4LRFSYJESODU:l/BFJDVRPVBJOLNQQGPCZH6V3B4B:l/PCAVIKPKGVV2R3QC6LVAUHQL5K:l/Z2L2NKG4ZXYK56XLSHDH4RFRXV:l/PZM57C3GVDXGRQS7PK7K4AVUM6:l/QBPTKHSWVJXL7TWHN5OQ2EHNNP:l/S5367OB7IR5VXRDDGODFLQWKX3:l/267KQMWEPLGOWGC67ZENWRZVG5:l/6YQNUXFRJPY72T7NWW2KNQTOYE,upperdir=31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/diff,workdir=31545c7f35faa5824ccca2adc0dd968f6f4e653e11184e0d3783f2b1bc977050/work 0 0
overlay /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged overlay rw,relatime,lowerdir=l/XWEPIUFAPP7ZI7GICWFLWYR2ZV:l/R5PG7LY6KPIUEYEZIQFPM55HZL:l/2YYFFSQILMJNGDI2EY64BA3DCK:l/M6FZJXQE2J3O4QLIF3Z6OV3IYV:l/OAPL6OCFMDPJSCFK5H5OKUZBB2:l/TVOOCBC2GTB3FRR3MMU6NAKDYF:l/HGVPR35DTAXOOIWH74Y4BSOUCK:l/4LMQBXJDIVIK2D5DI3WW473ZXZ,upperdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/diff,workdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/work 0 0
overlay /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged overlay rw,relatime,lowerdir=l/XWEPIUFAPP7ZI7GICWFLWYR2ZV:l/R5PG7LY6KPIUEYEZIQFPM55HZL:l/2YYFFSQILMJNGDI2EY64BA3DCK:l/M6FZJXQE2J3O4QLIF3Z6OV3IYV:l/OAPL6OCFMDPJSCFK5H5OKUZBB2:l/TVOOCBC2GTB3FRR3MMU6NAKDYF:l/HGVPR35DTAXOOIWH74Y4BSOUCK:l/4LMQBXJDIVIK2D5DI3WW473ZXZ,upperdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/diff,workdir=ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/work 0 0
proc /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/dev tmpfs rw,nosuid,mode=755 0 0
devpts /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
mqueue /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
sysfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
proc /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_proc proc ro,relatime 0 0
systemd-1 /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=0,timeout=0,minproto=5,maxproto=5,direct 0 0
binfmt_misc /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sysfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys sysfs ro,relatime 0 0
securityfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
pstore /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
debugfs /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/kernel/debug debugfs rw,relatime 0 0
fusectl /rootfs/var/lib/docker/overlay2/ec10c2ddd83d179003d5de17a4ec5e7b1a533c63d3e9e88f479e746b43f3af5b/merged/host_sys/fs/fuse/connections fusectl rw,relatime 0 0
shm /rootfs/var/lib/docker/containers/abdd6c4e1bf9e63bf6afab17913d1b7c434d32df0659163a16e6ca4cbfa8b346/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /rootfs/var/lib/docker/containers/9e1da532483b0632fccf489931520a08f1833eda2b05d6394e29d507a0977c31/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /rootfs/var/lib/docker/containers/c7e7945e69fc410589bd20196b87dd1ffc707de2fe8f6f0c177b71e16f6fb029/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
shm /rootfs/var/lib/docker/containers/dc101ac35f7edc560434c55d9485c00b832c47bdbfa4035f9af37b41e5dd0133/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/vda1 /etc/resolv.conf ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
/dev/vda1 /etc/hostname ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
/dev/vda1 /etc/hosts ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc /proc/bus proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/fs proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/irq proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,mode=755 0 0
tmpfs /proc/timer_stats tmpfs rw,nosuid,mode=755 0 0
tmpfs /proc/sched_debug tmpfs rw,nosuid,mode=755 0 0

But, as you may see, I've configured root proc with 

      - '-collector.procfs=/host_proc'
      - '-collector.sysfs=/host_sys'

About privileged mode - I do not see anything which may be useful in the code, it simply read /proc/mounts file, and apparently it doesn't do it right. This is something local, because we have around 15 nodes with incorrect mode, while rest 2500 nodes are OK.

Evertvz commented 7 years ago

Hello,

I would like to mention that I have the same behaviour on two of my systems. One is an Ubuntu 14.04 and another one Ubuntu 16.04. Both times the / is RO. The mounts command show that the filesystem is remounted RO due to errors but node-exporter claims all is fine.

We run node-exporter directly on the system, not in docker.

FYI I updated from node-exporter 0.12 to 0.14 but the behaviour remains the same.

SuperQ commented 7 years ago

@Evertvz would you please attach a copy of /proc/mounts from that system?

SuperQ commented 7 years ago

Ahh, it turns out I'm wrong about the /proc/mounts. We're getting the readonly status from a Statfs() call.

I tested this against my local machine and was unable to reproduce the problem (I created a read-only tmpfs mount).

onorua commented 7 years ago

hm... isn't it better to read /proc/mounts?

SuperQ commented 7 years ago

That's a good question, the syscall in theory should be good enough to get the filesystem status. I will have to research this a bit.

SuperQ commented 7 years ago

@Evertvz One interesting point would be to compare /proc/mounts with the node_exporter's PID /proc/$PID/mounts. There may still be some namspacing going on.

Evertvz commented 7 years ago

Hello @SuperQ ,

I'll blame it on the lack of sleep because it seems I was looking at the wrong place for the readonly fs. There is no problem with our / filesystem, it only states that it will remount ro if there are errors. There are some errors on our /home though, but this did not put the filesystem in readonly.

So in short, no issues with node-exporter from my side. The problem was located between the chair and the keyboard.

matthiasr commented 7 years ago

Ah, but the initial report was in Docker. Obviously, a syscall will not honor -collector.procfs=/host_proc. Not sure how to deal with this.

On Fri, Jun 23, 2017, 09:45 Evert notifications@github.com wrote:

Hello @SuperQ https://github.com/superq ,

I'll blame it on the lack of sleep because it seems I was looking at the wrong place for the readonly fs. There is no problem with our / filesystem, it only states that it will remount ro if there are errors. There are some errors on our /home though, but this did not put the filesystem in readonly.

So in short, no issues with node-exporter from my side. The problem was located between the chair and the keyboard.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/prometheus/node_exporter/issues/591#issuecomment-310596452, or mute the thread https://github.com/notifications/unsubscribe-auth/AAICBkdnVPKwklbee5JzkH6psp83EkHAks5sG20ogaJpZM4NtVa0 .

grobie commented 7 years ago

Don't run node_exporter in a docker container.

On Fri, Jun 23, 2017 at 9:55 AM Matthias Rampke notifications@github.com wrote:

Ah, but the initial report was in Docker. Obviously, a syscall will not honor -collector.procfs=/host_proc. Not sure how to deal with this.

On Fri, Jun 23, 2017, 09:45 Evert notifications@github.com wrote:

Hello @SuperQ https://github.com/superq ,

I'll blame it on the lack of sleep because it seems I was looking at the wrong place for the readonly fs. There is no problem with our / filesystem, it only states that it will remount ro if there are errors. There are some errors on our /home though, but this did not put the filesystem in readonly.

So in short, no issues with node-exporter from my side. The problem was located between the chair and the keyboard.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub < https://github.com/prometheus/node_exporter/issues/591#issuecomment-310596452 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AAICBkdnVPKwklbee5JzkH6psp83EkHAks5sG20ogaJpZM4NtVa0

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/prometheus/node_exporter/issues/591#issuecomment-310598326, or mute the thread https://github.com/notifications/unsubscribe-auth/AAANaAR7sd2VD3WwJSkjzkRFRIgb4jS_ks5sG29egaJpZM4NtVa0 .

SuperQ commented 7 years ago

@matthiasr We could possibly try and change the mount namespace in the container to the root mount namespace. We would need a nsenter type utility in the Docker package, or have an option to call setns inside the node_exporter.

onorua commented 7 years ago

@grobie it is easier say than accomplish. @SuperQ According to docker: https://github.com/rootfs/docker/commit/0ca94ce299faa7126668d24d5a5eadcbcc0d9a2b#diff-6ce9e79ddb91a3f06352abe1f2c72ecbR410

–hostns=true should do the trick, but I'm not sure about this option availability in k8s.

IMO, file based approach sounds like more reliable in this particular case.

SuperQ commented 7 years ago

@onorua --hostns=true does not exist in Docker, that is just a patch someone wrote.

grobie commented 7 years ago

@onorua It's easier than installing docker, you don't even need root. You installed docker on your system, so you can install node_exporter on your system. Systems have been operated for decades without Docker containers and there are countless supervisors available.

The node_exporter main use case is to monitor the machine, wrapping it in a virtual environment in isolated namespaces is asking for trouble.

I disagree that it's more reliable to try parsing a text file than to use a well defined system call.

matthiasr commented 7 years ago

But it does mean running a supervisor and getting the binaries delivered in the first place. At least on container focussed systems like CoreOS and RancherOS this is not a trivial task. @fabxc do you know of any plans to include the node exporter in CoreOS?

I don't know much about all the namespacing. Could a node exporter running in a privileged container detect the namespaces that it's in and "escape"? How does cAdvisor do this?

fabxc commented 7 years ago

No plans to my knowledge. @brancz is currently looking into fixing some of the oddnesses that come from running it in a container.

brancz commented 7 years ago

As a matter of fact we just did this for the net namespace. This is possible for all namespaces. For a start we did this to maintain the ability to use Pod networking, but collect metrics of the host net namespace. Basically what we're doing is:

We wrote a thin wrapper wrapper which written in rust as you cannot nsenter reliably in go because of the way goroutines/threading works.

Technically this can be done with any namespace, we could nsenter into all namespace under /host/proc/1/ns/*. The wrapper does need to run as root though as nsenter is only possible with root.

wtip commented 7 years ago

I'm running node exporter 0.13.0 as root on Ubuntu systems and not in any container. This morning we had an incident where a virtual host lost access to a storage array temporarily and one of the VMs remounted the filesystem RO. After sorting out the issue, rebooting the system and fsck'ing the filesystem, I checked our prometheus server for node_filesystem_readonly != 0 over the last few days and it doesn't return anything. The node exporter continued to run after the system remounted the / FS RO and other metrics were being reported.

I'm wondering if there's a bug with the implementation. Should this metric get updated if the filesystem remounts after the exporter was started?

matthiasr commented 7 years ago

Yes, it should.

On Mon, Nov 20, 2017, 21:09 William notifications@github.com wrote:

I'm running node exporter 0.13.0 as root on Ubuntu systems and not in any container. This morning we had an incident where a virtual host lost access to a storage array temporarily and one of the VMs remounted the filesystem RO. After sorting out the issue, rebooting the system and fsck'ing the filesystem, I checked our prometheus server for node_filesystem_readonly != 0 over the last few days and it doesn't return anything. The node exporter continued to run after the system remounted the / FS RO and other metrics were being reported.

I'm wondering if there's a bug with the implementation. Should this metric get updated if the filesystem remounts after the exporter was started?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/prometheus/node_exporter/issues/591#issuecomment-345815796, or mute the thread https://github.com/notifications/unsubscribe-auth/AAICBhsst5fHaN8yic_412obaBVHcT8Hks5s4dyFgaJpZM4NtVa0 .

michellevalentinova commented 6 years ago

Got the same with v0.16.0-rc.0 on Amazon Linux AMI 2017.09.

I decided to mount with -o errors=panic rather than the default read only which makes it harder to troubleshoot things.

discordianfish commented 6 years ago

It looks like the OP actually had no issue and just misread something, then the issues continues to talk about running in or outside docker, followed by two newer issue which might or might not be an real issue. This is confusing :) Let's close this issue here to avoid further confusion.

@wtip Do you still have this issue with 0.16.0? @supermch What exact issue do you have? node-exporter shows RW but it's RO right now? Or does it still show RW when the filesystem was remounted while node-exporter runs? Either way, can you please open a new issue for that?

davidbirdsong commented 6 years ago

Ok, so I think I have an actual issue. No docker, no VM, bare metal hardware.

OS: 

birdsong@fsba3 ~ $ cat /etc/redhat-release
CentOS release 6.7 (Final)

somewhat old kernel:

birdsong@fsba3 ~ $ uname -a
Linux fsba3.prod.foursquare.com 2.6.32-573.1.1.el6.x86_64 #1 SMP Sat Jul 25 17:05:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

node_exporter:

node_exporter, version 0.15.2 (branch: HEAD, revision: 98bc64930d34878b84a0f87dfe6e1a6da61e532d)
  build user:       birdsong@ezej8.prod.foursquare.com
  build date:       20180326-22:24:19
  go version:       go1.8.3

readonly filesystem:

birdsong@fsba3 ~ $ grep hdc3 /proc/mounts
/dev/sdc3 /export/hdc3 ext4 ro,noatime,nodiratime,barrier=1,data=ordered 0 0

borrowed/inspired by node_exporter: 

package main

import (
    "fmt"
    "syscall"
)

func main() {

    buf := new(syscall.Statfs_t)
    err := syscall.Statfs("/export/hdc3", buf)
    if err != nil {
        panic(err)
    }
    fmt.Printf("raw flags: %v\n", buf.Flags)
    fmt.Printf("anded w/ readonly mask: %v\n", buf.Flags&0x1)
}

output of program:

raw flags: 3104
anded w/ readonly mask: 0

I'm hunting for the value of ST_RDONLY, but perhaps a more flexible discovery of this flag value could help?

This stack overflow thread does not inspire much confidence that such a thing is reliably knowable.

onorua commented 6 years ago

@discordianfish I actually had the issue, and node_exporter showed wrong data, and it was clearly stated in the thread that the problem is because node_exporter is using syscall instead of reading /proc filesystem directly. Where did you get that I misread something?

grobie commented 6 years ago

I have also seen that the metric reporting a fs not in ro mode while it actually was.

discordianfish commented 6 years ago

@onorua Sorry, confused you with @Evertvz

discordianfish commented 6 years ago

So this isn't only an issue in docker, right? If it would, maybe it's due to the bindmount not being read-only while the actual filesystem is. In general the syscall should work in Docker since it rans against the bind-mount which is visible in the container.

davidbirdsong commented 6 years ago

The way I see it is it's either a weakness in syscall.Statfs or the hardcoded flag that node_exporter uses to suss out whether ST_RDONLY is set . I couldn't find another flag value to try out.

Allex1 commented 6 years ago

I have the same issue with node_exporter 0.15.2 on ec2 instance, centos6, 4.9.6 kernel. Disk is ro (from /proc/mounts) and node_filesystem_readonly metric is 0 for multiple disks/machines

Allex1 commented 6 years ago

I've done a couple of tests with both 0.15.2 and 0.16.0 and found out that:

As others mentioned probably reading /proc/mounts is a more reliable way of getting the mount status.

Hope this helps.

SuperQ commented 6 years ago

@Allex1 Thanks, that helps.

I think we should switch to parsing /proc/mounts for RO status. @discordianfish @grobie Any objection?

discordianfish commented 6 years ago

@SuperQ SGTM