Metrics Agent not detecting dynamic EBS mount

[tehmoon]

Hi!

I'm using this project in order to monitor all of our AWS servers the same way.

We have some EKS self-managed nodes deployed and I was interested in grabbing the disk size of any EBS volumes that would be mounted on a node. It appears that cloudwatch agent does not see the volumes and thus does not export it to cloudwatch metrics. This is deploying on https://aws.amazon.com/marketplace/pp/B07M68CJS5?ref_=srh_res_product_title which is hardened amazon-linux-2 instance.

I tried to read through the code but could not understand where the logic that grabs the mount points. I ended up reading gopsutil which I suspect it is what is used here but nothing that could indicate that specific volumes are filtered.

This is an example of my configuration specific to metrics:

     "disk": {
        "measurement": [
          "used_percent"
        ],
        "drop_device": true,
        "resources": [
          "*"
        ],
        "ignore_file_system_types": [
          "sysfs",
          "devtmpfs",
          "tmpfs"
        ]
      }
    },

And it works great, I do see the % used in cloudwatch every minute. But I do not see the other mount points. Specifically:

# grep nvme /proc/self/mountinfo
59 0 259:1 / / rw,noatime shared:1 - xfs /dev/nvme0n1p1 rw,attr2,inode64,noquota
863 59 259:1 /var/lib/kubelet/pods/f2e4450b-f54a-4828-9d65-709eb3d47731/volumes/kubernetes.io~configmap/prometheus-config/..2021_04_15_20_58_12.954665680/prometheus.yml /var/lib/kubelet/pods/f2e4450b-f54a-4828-9d65-709eb3d47731/volume-subpaths/prometheus-config/prometheus/1 rw,noatime shared:1 - xfs /dev/nvme0n1p1 rw,attr2,inode64,noquota
1132 59 259:3 / /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-0f38122ac3fb9338d rw,relatime shared:188 - ext4 /dev/nvme1n1 rw,data=ordered
1182 59 259:3 / /var/lib/kubelet/pods/7bf88b2b-fc75-4fdd-8aa2-5d74a98a5e17/volumes/kubernetes.io~aws-ebs/pvc-9487c72d-42d2-4079-b804-61f45b0e423e rw,relatime shared:188 - ext4 /dev/nvme1n1 rw,data=ordered
2154 59 259:6 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e61cc91c-2d71-4ee5-adbf-ca2a64006293/globalmount rw,relatime shared:253 - ext4 /dev/nvme4n1 rw,data=ordered
2159 59 259:6 / /var/lib/kubelet/pods/02e68d50-5a2f-4442-8be7-cac7b8d610e2/volumes/kubernetes.io~csi/pvc-e61cc91c-2d71-4ee5-adbf-ca2a64006293/mount rw,relatime shared:253 - ext4 /dev/nvme4n1 rw,data=ordered
2164 59 259:4 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-37fc4d8f-0a28-47a4-9b06-f8e9198d4e4c/globalmount rw,relatime shared:256 - ext4 /dev/nvme2n1 rw,data=ordered
2169 59 259:4 / /var/lib/kubelet/pods/ea168af4-c1b5-48fe-9918-7b005c8d66ee/volumes/kubernetes.io~csi/pvc-37fc4d8f-0a28-47a4-9b06-f8e9198d4e4c/mount rw,relatime shared:256 - ext4 /dev/nvme2n1 rw,data=ordered
2200 59 259:5 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3cbd32f0-abab-4cde-81cc-dfed92188fe0/globalmount rw,relatime shared:259 - ext4 /dev/nvme3n1 rw,data=ordered
2205 59 259:5 / /var/lib/kubelet/pods/17c774a1-0886-4eb9-85c2-0d1f1eff0483/volumes/kubernetes.io~csi/pvc-3cbd32f0-abab-4cde-81cc-dfed92188fe0/mount rw,relatime shared:259 - ext4 /dev/nvme3n1 rw,data=ordered
2221 59 259:7 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ec14585-5d1d-4d35-b896-375660bf6541/globalmount rw,relatime shared:268 - ext4 /dev/nvme5n1 rw,data=ordered
2263 59 259:7 / /var/lib/kubelet/pods/3c1e612f-014b-483c-af16-a102beaf025a/volumes/kubernetes.io~csi/pvc-8ec14585-5d1d-4d35-b896-375660bf6541/mount rw,relatime shared:268 - ext4 /dev/nvme5n1 rw,data=ordered
2505 59 259:7 /postgres /var/lib/kubelet/pods/3c1e612f-014b-483c-af16-a102beaf025a/volume-subpaths/pvc-8ec14585-5d1d-4d35-b896-375660bf6541/db/0 rw,relatime shared:268 - ext4 /dev/nvme5n1 rw,data=ordered
2640 59 259:4 /postgres /var/lib/kubelet/pods/ea168af4-c1b5-48fe-9918-7b005c8d66ee/volume-subpaths/pvc-37fc4d8f-0a28-47a4-9b06-f8e9198d4e4c/db/0 rw,relatime shared:256 - ext4 /dev/nvme2n1 rw,data=ordered
2799 59 259:8 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3d7dd169-f9fc-4050-8df7-7f1dc1df0415/globalmount rw,relatime shared:307 - ext4 /dev/nvme6n1 rw,data=ordered
2804 59 259:8 / /var/lib/kubelet/pods/ff2e7af6-bcc0-4985-822f-280636a48a07/volumes/kubernetes.io~csi/pvc-3d7dd169-f9fc-4050-8df7-7f1dc1df0415/mount rw,relatime shared:307 - ext4 /dev/nvme6n1 rw,data=ordered

I'm grabbing /proc/self/mountinfo as an example here because that's what I read in the gopsutil project.

Those mount points are done from the aws ebs daemonsets on kubernetes and are not part of the /etc/fstab file because they come and go.

I know that I can grab the same metrics using other tools, but I chose this tool to be deployed everywhere as it works out of the box for cloudwatch.

Thank you!

[tehmoon]

Actually, it worked when I used root instead of cwagent! Not so sure what's up with that, if I have more time, will investigate.

[jhnlsn]

Closing this issue as it's gone stale. Please reopen if you still are having an issue.