balena-io-modules / device-diagnostics

on-device diagnostics tool
9 stars 8 forks source link

Check if device is consuming close to max file descritors or inodes #279

Open 20k-ultra opened 3 years ago

20k-ultra commented 3 years ago

Add a diagnostic that will tell if the device is accessing too many files which causes other services to behaviour eradicate as they fail to open the files they need.

We can check file descriptors available with:

[root@box proc]# cat /proc/sys/fs/file-nr
3853    908     53182
|       |       |
|       |       |
|       |       max: maximum open file descriptors
|       free: total free allocated file descriptors
allocated: total allocated file descriptors since boot

To calculate the number that are currently in use, just do allocated - free. You could also calculate a percentage of used descriptors by doing ((allocated - free) / max) * 100

from https://stackoverflow.com/questions/7976769/getting-count-of-current-used-file-descriptors-from-c-code

and for inodes:

root@5795309:~# df -i
Filesystem                      Inodes  IUsed   IFree IUse% Mounted on
devtmpfs                        458076    415  457661    1% /dev
tmpfs                           492276     15  492261    1% /tmp
/dev/mmcblk0p2                   39936   6291   33645   16% /mnt/sysroot/active
/dev/disk/by-state/resin-state    5136     78    5058    2% /mnt/state
overlay                          39936   6291   33645   16% /
/dev/mmcblk0p6                 2863104 259005 2604099   10% /mnt/data
tmpfs                           492276      1  492275    1% /dev/shm
tmpfs                           492276    661  491615    1% /run
tmpfs                           492276     16  492260    1% /sys/fs/cgroup
/dev/mmcblk0p1                       0      0       0     - /mnt/boot
tmpfs                           492276     46  492230    1% /var/volatile
/dev/mmcblk0p3                   39936   6280   33656   16% /mnt/sysroot/inactive

Both of these could be combined into a single boolean: limited_file_access which is described as "This warning indicates if process on the device may be experiencing issues accessing files due to low inode or file descriptors available.