masonr / yet-another-bench-script

YABS - a simple bash script to estimate Linux server performance using fio, iperf3, & Geekbench
Do What The F*ck You Want To Public License
4.22k stars 430 forks source link

Detect if running on ZFS, in which case check for spa_asize_inflation value and multiply with workload size. Warn if larger than available disk space. #13

Closed webdock-io closed 3 years ago

webdock-io commented 4 years ago

ZFS datasets will see their performance crumble and be reported up to 10x slower than what they really are in FIO tests if the FIO workload multiplied by 24 is (on a default setup) larger than available disk space. See:

https://github.com/openzfs/zfs/issues/1668

This is "by design" and will not change, it seems. This means essentially that when yabs runs on a zfs backed system (say, an LXD container like we provide) the 2GB workload will be multiplied by 24 yielding 48GB free space required by ZFS in order to run at true/full throttle speeds.

To detect if on ZFS:

# df -Th
Filesystem             Type      Size  Used Avail Use% Mounted on
lxd/containers/testyabs zfs        47G  2.5G   45G   6% /

To grab the spa_asize_inflation value:

# cat /sys/module/zfs/parameters/spa_asize_inflation
24

So:

  1. Detect if on ZFS
  2. If so multiply workload (2GB) by spa_asize_inflation
  3. If larger than free disk space, warn user

Detecting this scenario and warning the user that they should get more free disk space, otherwise the test results are erroneous, would be good.

webdock-io commented 4 years ago

Sidenote, here is an example of a system with too little disk space and then enough disk space to run the FIO test according to the system spa_asize_inflation value:

Before:

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 13.81 MB/s    (3.4k) | 153.51 MB/s   (2.3k)
Write      | 13.83 MB/s    (3.4k) | 154.32 MB/s   (2.4k)
Total      | 27.64 MB/s    (6.9k) | 307.83 MB/s   (4.8k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 204.47 MB/s    (399) | 189.50 MB/s    (185)
Write      | 215.33 MB/s    (420) | 202.12 MB/s    (197)
Total      | 419.80 MB/s    (819) | 391.62 MB/s    (382)

After:

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 119.57 MB/s  (29.8k) | 1.95 GB/s    (30.5k)
Write      | 119.89 MB/s  (29.9k) | 1.96 GB/s    (30.7k)
Total      | 239.46 MB/s  (59.8k) | 3.92 GB/s    (61.3k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 1.50 GB/s     (2.9k) | 2.44 GB/s     (2.3k)
Write      | 1.58 GB/s     (3.0k) | 2.61 GB/s     (2.5k)
Total      | 3.08 GB/s     (6.0k) | 5.05 GB/s     (4.9k)
masonr commented 3 years ago

Resolved in PR #20 and merged in commit bf01122

Thanks again for the help tackling this one.