search

scottchiefbaker / dool

Python3 compatible fork of dstat
GNU General Public License v3.0
315 stars 58 forks source link

IO stats are inflated by ~10x #46

Closed KJ7LNW closed 10 months ago

KJ7LNW commented 10 months ago
SUMMARY

Dool shows high MB/s rates for disks, but dstat and iostat agree

ISSUE TYPE
DOOL VERSION
Dool 1.2.0
Written by Scott Baker <scott@perturb.org>
Forked from Dstat written by Dag Wieers <dag@wieers.com>
Homepage at https://github.com/scottchiefbaker/dool/

Platform posix/linux
Kernel 5.15.0-7.86.6.1.el9uek.x86_64-TEST+
Python 3.9.16 (main, May 29 2023, 00:00:00) 
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4.3.0.1)]

Terminal type: xterm-256color (color support)
Terminal size: 64 lines, 116 columns

Processors: 32
Pagesize: 4096
Clock ticks per secs: 100

internal:
    aio,cpu,cpu-adv,cpu-use,disk,epoch,fs,int,io,ipc,load,lock,mem,mem-adv,net,page,proc,raw,
    socket,swap,sys,tcp,time,udp,unix,vm,vm-adv,zones
/usr/share/dool:
    battery,battery-remain,bond,condor-queue,cpufreq,dbus,disk-avgqu,disk-avgrq,disk-svctm,disk-tps,
    disk-util,disk-wait,dool,dool-cpu,dool-ctxt,dool-mem,fan,freespace,fuse,gpfs,gpfs-ops,
    helloworld,ib,innodb-buffer,innodb-io,innodb-ops,jvm-full,jvm-vm,lustre,md-status,memcache-hits,
    mongodb-conn,mongodb-mem,mongodb-opcount,mongodb-queue,mongodb-stats,mysql-io,mysql-keys,
    mysql5-cmds,mysql5-conn,mysql5-innodb,mysql5-innodb-basic,mysql5-innodb-extra,mysql5-io,mysql5-keys,
    net-packets,nfs3,nfs3-ops,nfsd3,nfsd3-ops,nfsd4-ops,nfsstat4,ntp,postfix,power,proc-count,
    qmail,redis,rpc,rpcd,sendmail,snmp-cpu,snmp-load,snmp-mem,snmp-net,snmp-net-err,snmp-sys,
    snooze,squid,test,thermal,top-bio,top-bio-adv,top-childwait,top-cpu,top-cpu-adv,top-cputime,
    top-cputime-avg,top-int,top-io,top-io-adv,top-latency,top-latency-avg,top-mem,top-oom,utmp,
    vm-cpu,vm-mem,vm-mem-adv,vmk-hba,vmk-int,vmk-nic,vz-cpu,vz-io,vz-ubc,wifi,zfs-arc,zfs-l2arc,
    zfs-zil
DSTAT VERSION (PCP version for comparison, I know this one is not dool):
pcp-dstat 6.0.1
Written by the PCP team <pcp@groups.io> and Dag Wieers <dag@wieers.com>
Homepages at https://pcp.io/ and http://dag.wieers.com/home-made/dstat/

Platform Linux
Kernel 5.15.0-7.86.6.1.el9uek.x86_64-TEST+
Python 3.9.16 (main, May 29 2023, 00:00:00) 
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4.3.0.1)]

Terminal type: xterm-256color (color support)
Terminal size: 64 lines, 116 columns

Processors: 32
Pagesize: 4096
Clock ticks per second: 100

timestamp plugins:
    epoch, epoch-adv, time, time-adv
/etc/pcp/dstat plugins:
    aio, battery, cpu, cpu-adv, cpu-use, disk, disk-avgqu, disk-avgrq, disk-svctm, disk-tps, disk-util, 
    disk-wait, dm, dm-avgqu, dm-avgrq, dm-svctm, dm-tps, dm-util, dm-wait, entropy, freespace, fs, gpfs, 
    gpfs-ops, innodb-buffer, innodb-io, innodb-ops, int, io, ipc, load, lock, md, md-avgqu, md-avgrq, 
    md-svctm, md-tps, md-util, md-wait, mem, mem-adv, memcache, mongodb-conn, mongodb-dbstats, 
    mongodb-latency, mongodb-mem, mongodb-opcount, mysql-io, mysql-keys, net, net-packets, nfs3, nfs3-ops, 
    nfs4, nfs4-ops, nfsd3, nfsd3-ops, nfsd4, nfsd4-ops, page, part, part-avgqu, part-avgrq, part-svctm, 
    part-tps, part-util, part-wait, postfix, proc, proc-count, raw, redis, redis-client, redis-mem, rpc, rpcd, 
    socket, socket6, swap, sys, tcp, top-bio, top-bio-adv, top-childwait, top-cpu, top-cpu-adv, top-cputime, 
    top-cputime-avg, top-io, top-io-adv, top-latency, top-latency-avg, top-mem, top-oom, udp, unix, utmp, vm, 
    vm-adv, zfs-arc, zfs-l2arc, zfs-zil
OS / ENVIRONMENT

Oracle Linux 9

STEPS TO REPRODUCE
/bin/dool -D nvme0n1,nvme1n1,sda -d 1 | while read a; do echo "dool  : $a"; done & 
/bin/dstat -D nvme0n1,nvme1n1,sda -d 1 | while read a; do echo "dstat : $a"; done & 
iostat /dev/nvme[01]n* /dev/sda 1 | while read a; do echo "iostat: $a"; done &

# wait a while and then:
kill %1 %2 %3
EXPECTED RESULTS

We expect what iostat and dstat show:

iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 3.67    0.00    2.60    2.10    0.00   91.64
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         715.00         0.00      5830.00         0.00          0       5830          0
iostat: nvme1n1         716.00         0.00      5830.50         0.00          0       5830          0
iostat: sda               0.00         0.00         0.00         0.00          0          0          0
iostat: 
dstat : 0  5920k:   0  5936k:   0     0
ACTUAL RESULTS

as above, but also mixed into the output, dool shows 48M/s iinstead of ~5M/s:

...
dool  : 0    48M┊   0    48M┊   0     0
...

Here is a longer run. Note that iostat and dstat numbers agree, but dool's numbers are much bigger than they should be:

# /bin/dool -D nvme0n1,nvme1n1,sda -d 1 | while read a; do echo "dool  : $a"; done & 
/bin/dstat -D nvme0n1,nvme1n1,sda -d 1 | while read a; do echo "dstat : $a"; done & 
iostat /dev/nvme[01]n* /dev/sda 1 | while read a; do echo "iostat: $a"; done &
[1] 2786885
[2] 2786887
[3] 2786889
0 [root@hv1:~]# iostat: Linux 5.15.0-7.86.6.1.el9uek.x86_64-TEST+ (hv1.ewheeler.net)    08/31/2023  _x86_64_    (32 CPU)
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 0.52    0.00    0.75    0.31    0.00   98.42
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         625.23       581.21      7943.58         0.00  318228259 4349328304          0
iostat: nvme1n1         640.32      1039.38      7943.58         0.00  569087923 4349328304          0
iostat: sda              89.51       131.62      3074.69         0.00   72065292 1683477916          0
iostat: 
dool  : dsk/nvme0n1 dsk/nvme1n1 ┄┄dsk/sda┄┄
dool  : read  writ  read  writ  read  writ
dool  : 4761K   65M┊8515K   65M┊1078K   25M
dstat : dsk/nvme0n1 dsk/nvme1n1 --dsk/sda--
dstat : read  writ: read  writ: read  writ
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 3.82    0.00    1.69    1.50    0.00   92.99
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         640.00         0.00      3786.50         0.00          0       3786          0
iostat: nvme1n1         640.00         0.00      3786.50         0.00          0       3786          0
iostat: sda             115.00         0.00      6636.00         0.00          0       6636          0
iostat: 
dool  : 0    33M┊   0    33M┊   0    46M
dstat : 0  4391k:   0  4391k:   0   776k
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 5.83    0.00    3.13    1.76    0.00   89.28
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         790.00         0.00      5720.00         0.00          0       5720          0
iostat: nvme1n1         790.00         0.00      5720.00         0.00          0       5720          0
iostat: sda               1.00         4.00         0.00         0.00          4          0          0
iostat: 
dool  : 0    47M┊   0    47M┊  33K    0
dstat : 0  5723k:   0  5723k:4093B    0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 5.20    0.00    2.41    1.57    0.00   90.82
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         666.00       224.00      5972.50         0.00        224       5972          0
iostat: nvme1n1         667.00       496.00      5972.50         0.00        496       5972          0
iostat: sda               7.00        32.00       704.00         0.00         32        704          0
iostat: 
dool  : 1835K   50M┊4063K   50M┊ 262K 5767K
dstat : 224k 5993k: 496k 5993k:  32k  704k
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 10.06    0.00    4.26    1.25    0.00   84.43
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         666.00         0.00      5926.00         0.00          0       5926          0
iostat: nvme1n1         667.00         0.00      5926.00         0.00          0       5926          0
iostat: sda               0.00         0.00         0.00         0.00          0          0          0
iostat: 
dool  : 0    49M┊   0    49M┊   0     0
dstat : 0  5824k:   0  5824k:   0     0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 6.20    0.00    3.44    1.44    0.00   88.92
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         615.00         0.00      6146.00         0.00          0       6146          0
iostat: nvme1n1         615.00         0.00      6146.00         0.00          0       6146          0
iostat: sda               0.00         0.00         0.00         0.00          0          0          0
iostat: 
dool  : 4194K   49M┊2621K   49M┊   0  6816K
dstat : 512k 6145k: 320k 6145k:   0   832k
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 8.79    0.00    3.66    1.44    0.00   86.10
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         687.00       512.00     10642.50         0.00        512      10642          0
iostat: nvme1n1         690.00       352.00     10642.50         0.00        352      10642          0
iostat: sda               7.00         4.00       832.00         0.00          4        832          0
iostat: 
dool  : 0    87M┊ 262K   87M┊  33K    0
dstat : 0    10M:  32k   10M:4096B    0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 5.51    0.00    2.51    1.88    0.00   90.10
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         815.00         0.00      7039.50         0.00          0       7039          0
iostat: nvme1n1         817.00        32.00      7039.50         0.00         32       7039          0
iostat: sda               0.00         0.00         0.00         0.00          0          0          0
iostat: 
dool  : 0    57M┊ 262K   57M┊   0     0
dstat : 0  7039k:  32k 7039k:   0     0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 6.64    0.00    3.23    1.75    0.00   88.37
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         822.00         0.00      7152.50         0.00          0       7152          0
iostat: nvme1n1         825.00       160.00      7152.50         0.00        160       7152          0
iostat: sda               2.00        16.00       128.00         0.00         16        128          0
iostat: 
dool  : 0    59M┊1311K   59M┊ 131K 1049K
dstat : 0  7130k: 160k 7130k:  16k  128k
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 2.57    0.00    2.04    1.88    0.00   93.51
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         864.00         0.00      7575.50         0.00          0       7575          0
iostat: nvme1n1         865.00        16.00      7575.50         0.00         16       7575          0
iostat: sda               0.00         0.00         0.00         0.00          0          0          0
iostat: 
dool  : 0    64M┊ 131K   64M┊   0     0
dstat : 0  7717k:  16k 7717k:   0     0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 2.76    0.00    1.72    2.07    0.00   93.45
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         881.00         0.00      7219.00         0.00          0       7219          0
iostat: nvme1n1         885.00        48.00      7219.00         0.00         48       7219          0
iostat: sda               2.00         8.00         0.00         0.00          8          0          0
iostat: 
dool  : 0    58M┊ 393K   58M┊  66K    0
dstat : 0  7043k:  48k 7043k:8186B    0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 3.24    0.00    2.31    2.06    0.00   92.40
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         838.00       232.00     20725.00         0.00        232      20725          0
iostat: nvme1n1         838.00       152.00     20725.00         0.00        152      20725          0
iostat: sda              21.00         0.00      2528.00         0.00          0       2528          0
iostat: 
dool  : 1901K  170M┊1245K  170M┊   0    21M
dstat : 232k   20M: 152k   20M:   0  2528k
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 4.22    0.00    2.10    1.66    0.00   92.02
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         759.00         0.00      6157.00         0.00          0       6157          0
iostat: nvme1n1         759.00         0.00      6157.00         0.00          0       6157          0
iostat: sda               4.00        28.00         0.00         0.00         28          0          0
iostat: 
dool  : 0    49M┊   0    49M┊ 229K    0
dstat : 0  6092k:   0  6092k:  28k    0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 5.85    0.00    2.54    1.60    0.00   90.02
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         730.00       280.00      6146.00         0.00        280       6146          0
iostat: nvme1n1         731.00       376.00      6146.00         0.00        376       6146          0
iostat: sda               5.00         4.00       640.00         0.00          4        640          0
iostat: 
dool  : 2294K   51M┊3080K   51M┊  66K 5243K
dstat : 280k 5974k: 376k 5974k:8192B  640k
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 6.64    0.00    3.10    1.35    0.00   88.91
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         707.00         0.00      5616.50         0.00          0       5616          0
iostat: nvme1n1         707.00         0.00      5616.50         0.00          0       5616          0
iostat: sda               2.00         8.00         0.00         0.00          8          0          0
iostat: 
dool  : 0    46M┊   0    46M┊  33K    0
dstat : 0  6424k:   0  6424k:8192B    0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 12.35    0.00    4.17    2.04    0.00   81.44
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         826.00         0.00      7360.50         0.00          0       7360          0
iostat: nvme1n1         827.00        16.00      7360.50         0.00         16       7360          0
iostat: sda               5.00        56.00         0.00         0.00         56          0          0
iostat: 
dool  : 0    60M┊ 131K   60M┊ 459K    0
dstat : 0  6870k:  16k 6870k:  52k    0
iostat: 
iostat: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
iostat: 5.17    0.00    2.88    2.41    0.00   89.54
iostat: 
iostat: Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
iostat: nvme0n1         429.00         0.00      3466.50         0.00          0       3466          0
iostat: nvme1n1         433.00       160.00      3467.00         0.00        160       3467          0
iostat: sda             102.00         0.00      6364.00         0.00          0       6364          0
iostat: 
dool  : 0    26M┊1311K   26M┊   0    52M
dstat : 0  2753k: 160k 2753k:   0  6507k
KJ7LNW commented 10 months ago

In case it matters, the sda sector size is 4k, not 512:

# blockdev --getss /dev/sda
4096
# blockdev --getss /dev/nvme0n1 
512
# blockdev --getss /dev/nvme1n1
512
scottchiefbaker commented 10 months ago

dool defaults to show disk and network in bits per second. Try --bytes to get output in bytes if that's what you'd prefer.

KJ7LNW commented 10 months ago

Ahh! Ok, that explains it.

Bits-per-second makes sense for network, but that is unusual for a storage, and it deviates from both Dag's original dstat and RH's dstat replacement for disk IO stats.

What do you think about defaulting everything to bytes/sec except network?

scottchiefbaker commented 10 months ago

I use dool to watch my network interfaces which then results in disk read/write. If they were in different scales it would mess me up. I'm going to leave things as they are. Feel free to use --bytes if it works better in your environment.

KJ7LNW commented 10 months ago

Got it, thanks for the hint. I only mentioned changing the default to bytes because the README.md says "Dool is a Python3 compatible fork of Dstat."

IIRC, the original dstat uses bytes by default. Now that I know about --bytes, I'm fine either way, just not sure how strict you had planned to be in terms of dstat compatibility.

Thanks for your help!

scottchiefbaker commented 10 months ago

This comes up every so often, I should put a section in the README