[Feat]: Better diskstats for multiple-volume Btrfs filesystems

[Forza-tng]

Problem

Background

Netdata can use /dev/disk/by-label/<label> to monitor disk performance instead of plain /dev/sdX.

Let's use the filesystem with label btrfs-root as an example.

netdata.conf

[plugin:proc:/proc/diskstats:btrfs-root]
    enable = yes
    enable performance metrics = yes
    bandwidth = auto
    operations = auto
    merged operations = auto
    i/o time = auto
    queued operations = auto
    utilization percentage = auto
    extended operations = auto
    backlog = auto

This gives a nice graph with all the performance metrics, just as if it was a plain device. Great!

Problem

The problem is that the /dev/disk-by/<label> only points to one device. So when we have a filesystem with multiple devices, only one would be monitored by Netdata.

# ll /dev/disk/by-label/btrfs-root
lrwxrwxrwx 1 root root 10 Aug 23 13:16 /dev/disk/by-label/btrfs-root -> ../../sdb3

However when we use blkid |grep btrfs-root we clearly see two devices.

# blkid |grep btrfs-root
/dev/sdb3: LABEL="btrfs-root" UUID="558ec0b4-869e-4f8e-a143-258d4d380847" TYPE="btrfs"
/dev/sda3: LABEL="btrfs-root" UUID="558ec0b4-869e-4f8e-a143-258d4d380847" TYPE="btrfs"

This can also be verified with btrfs filesystem show

# btrfs filesystem show
Label: 'btrfs-root'  uuid: 558ec0b4-869e-4f8e-a143-258d4d380847
        Total devices 2 FS bytes used 12.30GiB
        devid    1 size 461.26GiB used 15.03GiB path /dev/sdb3
        devid    2 size 461.26GiB used 15.03GiB path /dev/sda3

Label: 'nas-hdd'  uuid: 7745e2f7-5c67-4b18-844b-8e93399f7b0b
        Total devices 6 FS bytes used 4.16TiB
        devid   11 size 16.37TiB used 1.40TiB path /dev/sdk1
        devid   12 size 16.37TiB used 1.40TiB path /dev/sdf1
        devid   13 size 16.37TiB used 1.40TiB path /dev/sdp1
        devid   14 size 16.37TiB used 1.40TiB path /dev/sdu1
        devid   15 size 12.73TiB used 1.39TiB path /dev/sdd1
        devid   16 size 12.73TiB used 1.39TiB path /dev/sdi1

Description

What I would like is that Netdata enumerates all devices that belong to the filesystem and use the combined stats (I guess from /proc/diskstats) in the charts.

I can see two options:

1. A combined chart with each device as a separate line (mockup)
2. The charts simple contain the sum of all device stats.

Just an important note. It is important to montor the partition usage and not the whole device since otherwise the values would include IOs made outside of the filesystem on those devices.

Importance

really want

Value proposition

1. It would allow much better monitorong of filesystems' performance metrics.
2. It would make monitoring easier to understand. ...

Proposed implementation

My first suggestion would be to use blkid or equivalent to find all devices matching the UUID of the filesystem. /sys/fs/btrfs or python-btrfs could be used as well.

[Forza-tng]

It is probably be better to use the /sys/fs/btrfs{fs_uuid} interface to enumerate filesystems and their member devices.

# ls /sys/fs/btrfs/fe0a1142-51ab-4181-b635-adbf9f4ea6e6/
drwxr-xr-x 6 root root    0 Aug 20 11:39 ./
drwxr-xr-x 8 root root    0 Aug 20 11:39 ../
drwxr-xr-x 5 root root    0 Aug 20 11:39 allocation/
lrwxrwxrwx 1 root root    0 Sep 14 20:32 bdi -> ../../../devices/virtual/bdi/btrfs-2/
-rw-r--r-- 1 root root 4096 Sep 14 20:32 bg_reclaim_threshold
-r--r--r-- 1 root root 4096 Sep 14 20:32 checksum
-r--r--r-- 1 root root 4096 Sep 14 20:32 clone_alignment
drwxr-xr-x 2 root root    0 Aug 20 13:11 devices/
drwxr-xr-x 4 root root    0 Sep 14 20:32 devinfo/
-r--r--r-- 1 root root 4096 Sep 14 20:32 exclusive_operation
drwxr-xr-x 2 root root    0 Aug 25 18:59 features/
-r--r--r-- 1 root root 4096 Sep 14 20:32 generation
-rw-r--r-- 1 root root 4096 Aug 20 13:12 label
-r--r--r-- 1 root root 4096 Sep 14 20:32 metadata_uuid
-r--r--r-- 1 root root 4096 Sep 14 20:32 nodesize
-rw-r--r-- 1 root root 4096 Sep 14 20:32 quota_override
-rw-r--r-- 1 root root 4096 Sep 14 20:32 read_policy
-r--r--r-- 1 root root 4096 Sep 14 20:32 sectorsize

label contains the filesystem label devices/ contains links to each block device

# ll devices/
total 0
drwxr-xr-x 2 root root 0 Aug 20 13:11 ./
drwxr-xr-x 6 root root 0 Aug 20 11:39 ../
lrwxrwxrwx 1 root root 0 Aug 20 13:11 sdb2 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:02:00.1/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb2/
lrwxrwxrwx 1 root root 0 Aug 20 13:11 sdd2 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:02:00.1/ata6/host5/target5:0:0/5:0:0:0/block/sdd/sdd2/

[Forza-tng]

Hi. Tried to make a bash collector for this, but can't quite understand how to assign labels to member devices.

I got a chart per filesystem like this:

But I'd like to be able to show all member devices so I can choose to display the totals per filesystem or per member. Maybe something like this:

The initial test script is this. It's very slow because it enumerate all devices over and over.

# shellcheck shell=bash

btrfsvol_update_every=1
btrfsvol_timeout=2
btrfsvol_priority=80000

basename_cmd="$(type -fp basename)"
readlink_cmd="$(type -fp readlink)"
DISKSTATS_FILE="/proc/diskstats"
sysfs_path="/sys/fs/btrfs"

## Which fs UUIDs to invlude in charts.
## To do:  use a config file instead
#UUIDS="fe0a1142-51ab-4181-b635-adbf9f4ea6e6 c08bb98b-3b98-4dbb-a7c0-5540c2af781b aa358efb-ce43-498c-9997-0d35ba13261f df68a30d-d26e-4b9c-9606-a130e66ce63d"
UUIDS=""
for entry in "$sysfs_path"/*; do
    if [ -d "$entry" ] && [[ "$($basename_cmd "$entry")" =~ ^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$ ]]; then
        uuid=$($basename_cmd "$entry")
        UUIDS+=" $uuid"
    fi
done

btrfsvol_check(){
    ## To do? verify UUIDs exists and are valid.
    return 0
}

_label(){
    echo "$(cat "/sys/fs/btrfs/${1}/label")"
}

btrfsvol_create(){
    local uuid
    local name
    # perhapd create an associative array for fs->dev mapping to avoid looking up every update.

    for uuid in ${UUIDS}; do
        name="$(_label $uuid)"
        # create the charts
        cat << EOF
CHART btrfsvol.ios.${name} '' "Btrfs filesystem IO" "IO/s" ios '' line $((btrfs_priority + 5)) $btrfs_update_every '' '' 'btrfsvol'
EOF
    done
}

btrfsvol_update(){
    local microseconds=$1
    local uuid 
    local name
    local line
    local -a fields
    #diskstats="$(< "/proc/diskstats")"

    for uuid in ${UUIDS}; do
        name="$(_label ${uuid})"

        # Initialize variables for statistics.
        TOTAL_READ_COMPLETED=0
        TOTAL_READ_MERGED=0
        TOTAL_READ_SECTORS=0
        TOTAL_READ_TIME=0
        TOTAL_WRITE_COMPLETED=0
        TOTAL_WRITE_MERGED=0
        TOTAL_WRITE_SECTORS=0
        TOTAL_WRITE_TIME=0
        read_completed=0
        read_merged=0
        read_sectors=0
        read_time=0
        write_completed=0
        write_merged=0
        write_sectors=0
        write_time=0

        echo "BEGIN btrfsvol.ios.${name} $microseconds"
        echo "DIMENSION reads '' incremental 1 1"
        echo "DIMENSION writes '' incremental -1 1"
        while read -r line; do
            # Split the line into fields.
            read -a fields <<< "$line"
            # Extract relevant fields.
            device_name="${fields[2]}"
            # Extract the actual device name from the symlink.
            symlink_path="/sys/fs/btrfs/${uuid}/devices/$device_name"

            actual_device_name=$($basename_cmd "$($readlink_cmd -n "$symlink_path")")

            # Check if the device belongs to the specified Btrfs filesystem.
            if [[ "$actual_device_name" == "$device_name" ]]; then

                ## format of DISKSTATS_FILE:
                # 8 112 sdh 3357 1049 63026 12452 3774216 53250 3634979608 28582109 0 5081312 28614242 0 0 0 0 4307 19680 
                read_completed="${fields[3]}"
                #read_merged="${fields[4]}"
                #read_sectors="${fields[5]}"
                #read_time="${fields[6]}"
                write_completed=${fields[7]}
                #write_merged=${fields[8]}
                #write_sectors=${fields[9]}
                #write_time="${fields[10]}"

                ## add individual drill-down labels per device
                # to do

                ## Update the total fs UUID statistics.
                TOTAL_READ_COMPLETED=$((TOTAL_READ_COMPLETED + read_completed))
                TOTAL_WRITE_COMPLETED=$((TOTAL_WRITE_COMPLETED + write_completed))
            fi
        done < "$DISKSTATS_FILE" 

        echo "SET reads = $TOTAL_READ_COMPLETED"
        echo "SET writes = $TOTAL_WRITE_COMPLETED"
        echo "END"

    done
}

What I'd like ultimately is per filesystem, with drill down or filter per member device:

• ios r/w
• bytes r/w

It would be great if this could be integrated with the existing Btrfs collector instead of as a separate one, but I'd be happy with the bash version too.