Monitoring with node-exporter

[edusouzaxx]

RancherOS Version: (ros os version)

v1.5.5

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)

Vmware

Why node-exporter does not take metrics from additional filesystems in RancherOS

RancherOS version

cat /etc/os-release

NAME="RancherOS"
VERSION=v1.5.5
ID=rancheros
ID_LIKE=
VERSION_ID=v1.5.5
PRETTY_NAME="RancherOS v1.5.5"
HOME_URL="http://rancher.com/rancher-os/"
SUPPORT_URL="https://forums.rancher.com/c/rancher-os"
BUG_REPORT_URL="https://github.com/rancher/os/issues"
BUILD_ID=

mounts configuration

ros config set mounts '[["/dev/sdb1","/var/lib/user-docker","ext4",""],["/dev/sdb2","/data","ext4",""],["/dev/sdb3","/var/lib/kubelet","ext4",""],["/dev/sdb4","/opt/rke","ext4",""]]'

filesystems

df -h 

Filesystem      Size  Used Avail Use% Mounted on
overlay         4.7G  1.9G  2.6G  43% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/sda1       4.7G  1.9G  2.6G  43% /mnt
none             16G  1.1M   16G   1% /run
shm              64M     0   64M   0% /dev/shm
/dev/sdb1        98G  600M   93G   1% /var/lib/user-docker
/dev/sdb2        49G   53M   47G   1% /data
/dev/sdb3       9.8G   37M  9.3G   1% /var/lib/kubelet
/dev/sdb4       9.8G   37M  9.3G   1% /opt/rke

node-exporter service configuration

node_exporter:
  image: prometheus/node-exporter:v0.18.1
  command: |
    --path.procfs=/host/proc
    --path.sysfs=/host/sys
    --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$$
    --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($$|/)
    --collector.ksmd
    --collector.filesystem
    --collector.mountstats
    --path.rootfs=/host/root
    --web.listen-address=0.0.0.0:9100
    --log.level=debug
  privileged: true
  labels:
    io.rancher.os.scope: "system"
    io.rancher.os.after: "docker"
    io.rancher.os.detach: "true"
  ports:
    - "9100:9100/tcp"
  volumes:
    - /proc:/host/proc:ro
    - /sys:/host/sys:ro
    - /:/host/root:ro
  restart: unless-stopped 

metrics

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP node_filesystem_free_bytes Filesystem free space in bytes.
TYPE node_filesystem_free_bytes gauge
node_filesystem_free_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 2.98029056e+09
node_filesystem_free_bytes{device="none",fstype="tmpfs",mountpoint="/run"} 1.6853835776e+10
100 57003    0 57003    0     0  1562k      0 --:--:-- --:--:-- --:--:-- 1637k

problem

I would like to receive metrics for the filesystems:

/data /var/lib/user-docker /var/lib/kubelet /opt/rke

but I only have metrics from filesystem:

/

[edusouzaxx]

The workaround I found:

Create the mountpoints under /mnt:

By default, /media and /mnt are mounted as shared in the console container. This means that mounts within these directories will propagate to the host as well as other system services that mount these folders as shared.

ros config set mounts '[["/dev/sdb1","/mnt/var/lib/docker","ext4",""],["/dev/sdb2","/mnt/data","ext4",""],["/dev/sdb3","/mnt/var/lib/kubelet","ext4",""],["/dev/sdb4","/mnt/opt/rke","ext4",""]]'

And create symbolic links on cloud-config like:

write_files:

path: /etc/rc.local
permissions: “0755”
owner: root
content: |
#!/bin/bash
ln -s /mnt/data /data
ln -s /mnt/var/lib/kubelet /var/lib/kubelet
ln -s /mnt/opt/rke /opt/rke