Monitor I/O error counters

[joonas-fi]

Problem

Would be nice to show a big fat "volume has serious problems" error in Varasto to alert users on time.

Solution

Look at Linux kernel's I/O error counters per block device.

How to do that?

Block device I/O errors

Status: BLOCKED 🚧

Seems to be close to impossible:

There is no error counter for block devices in Linux.

Source: https://medium.com/opsops/block-device-error-count-in-linux-f010ed4f7f76

No I/O error counters in the statistics kept by kernel

The Linux kernel knows when disk IO errors happen and it will tell you about them. But only in the kernel message log, in a format that can vary from disk type to disk type and changes over time.

Source: https://utcc.utoronto.ca/~cks/space/blog/linux/DiskErrorsSysfsCountWish

Other sources:

• https://stackoverflow.com/a/54118314
• https://docs.kernel.org/admin-guide/iostats.html kernel docs for I/O stats

Filesystem driver error counter

Some Linux filesystems do have some reporting of errors in sysfs, for example ext4's sysfs entries include an error count and information about the first and most recent errors experienced. But it's not clear what sort of 'errors' these are about, and the information is apparently persisted in the filesystem superblock instead of being transient. You might want to monitor the error count, though.

(Source)

I encountered an error. Here's from $ dmesg:

[Sat Jun  1 19:24:37 2024] sd 9:0:0:2: [sdd] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[Sat Jun  1 19:24:37 2024] sd 9:0:0:2: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 05 e1 84 08 00 00 00 00 08 00 00

But the corresponding error counters in the FS driver are big fat nil:

$ cat /sys/fs/ext4/sdd1/{errors_count,last_error_time}
0
0

Utterly useless.