Raid0 partition and single NVME disks that comprise that partition metrics don't match

[vladzcloudius]

Installation details Panel Name: Disk Writes/Reads Dashboard Name: OS Metrics Scylla-Monitoring Version: 4.7.1 Scylla-Version: 2024.1.3-0.20240401.64115ae91a55 Kernel version on all nodes: 5.15.0-1058-gcp

Description Throughputs (bytes or OPS) of the RAID0 volume (md0 in screenshots below) is supposed to be equal to a sum of corresponding values on physical disks comprising it. However it's far from it. In some cases, like in screenshots below, the corresponding value is even less. In the example below md0 is a RAID0 volume assembled from 4 NVMe disks: nvme0n1,2,3,4

Here is the screenshot showing md0 and only nvme0n1 from all nodes (but the same picture is on all other disks:

Here you can see the values from all disks on a single node clearly showing the problem:

I ran iostat on one of the node trying to see if this is maybe some kernel issue but no, iostat shows values that totally add up:

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
md0            323.00  99448.00     0.00   0.00    2.12   307.89    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.68   9.60
nvme0n1         86.00  24836.00     0.00   0.00    2.50   288.79    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.21   6.80
nvme0n2         75.00  23640.00     0.00   0.00    2.33   315.20    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.17   4.80
nvme0n3         80.00  24832.00     0.00   0.00    2.40   310.40    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.19   6.00
nvme0n4         82.00  26140.00     0.00   0.00    2.35   318.78    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.19   6.80
sda              0.00      0.00     0.00   0.00    0.00     0.00  128.00    736.00     3.00   2.29    0.57     5.75    0.00      0.00     0.00   0.00    0.00     0.00    0.07   1.60

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.03    0.00    1.60    0.00    0.00   96.37

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
md0            430.00 149624.00     0.00   0.00    2.17   347.96    1.00      4.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.93   8.80
nvme0n1         92.00  33124.00     0.00   0.00    2.70   360.04    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.25   7.60
nvme0n2         96.00  33804.00     0.00   0.00    2.41   352.12    1.00      4.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.23   8.40
nvme0n3         92.00  33256.00     0.00   0.00    2.41   361.48    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.22   6.80
nvme0n4        100.00  33056.00     0.00   0.00    2.19   330.56    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.22   6.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.37    0.00    1.31    0.00    0.00   96.32

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
md0            290.00  98304.00     0.00   0.00    3.52   338.98    4.00     56.00     0.00   0.00    0.00    14.00    0.00      0.00     0.00   0.00    0.00     0.00    1.02   6.80
nvme0n1         88.00  29924.00     0.00   0.00    3.08   340.05    1.00     32.00     0.00   0.00    0.00    32.00    0.00      0.00     0.00   0.00    0.00     0.00    0.27   5.60
nvme0n2         85.00  27560.00     0.00   0.00    2.79   324.24    2.00     16.00     0.00   0.00    0.50     8.00    0.00      0.00     0.00   0.00    0.00     0.00    0.24   6.00
nvme0n3         75.00  28412.00     0.00   0.00    2.77   378.83    1.00      8.00     0.00   0.00    0.00     8.00    0.00      0.00     0.00   0.00    0.00     0.00    0.21   5.60
nvme0n4         92.00  28792.00     0.00   0.00    2.63   312.96    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.24   5.20

We saw similar behavior on multiple clusters.

[vladzcloudius]

cc @tarzanek @vreniers @mkeeneyj

[amnonh]

@vladzcloudius if I get it right, this is a node_exporter issue, right?

[vladzcloudius]

@vladzcloudius if I get it right, this is a node_exporter issue, right?

Could be.

[amnonh]

@vladzcloudius could it be: https://github.com/prometheus/node_exporter/issues/2310