search

lsst-uk / somerville-operations

User issue reporting and tracking for the Somerville Cloud
0 stars 0 forks source link

NVMe Cinder Volume IO Tests #193

Open astrodb opened 2 months ago

astrodb commented 2 months ago

This issue is for tracking raw IO tests of the new NVMe Cinder volumes deployed on Somerville.

Test #1 - Linux tools - dd and fio Run on sv-qserv-nvme-worker7 OS: Ubuntu Jammy, Flavor: qserv-worker-v5

root@sv-qserv-nvme-worker-7:/qserv-data/test# dd if=/dev/zero of=local-test2 conv=fsync bs=512k count=2000 2000+0 records in 2000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 1.65408 s, 634 MB/s

root@sv-qserv-nvme-worker-7:/qserv-data/test# fio --name TEST --filename=temp.file --rw=write --size=64g --io_size=128g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=600

Run status group 0 (all jobs): WRITE: bw=1238MiB/s (1298MB/s), 1238MiB/s-1238MiB/s (1298MB/s-1298MB/s), io=128GiB (137GB), run=105884-105884msec

astrodb commented 2 months ago

Comparison results taken from Euclid cluster worker node, running bare metal on a 1.5TB NVMe:

[root@wulver1 tmp]# dd if=/dev/zero of=local-test2 conv=fsync bs=512k count=2000 2000+0 records in 2000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.39539 s, 438 MB/s

[root@wulver1 tmp]# fio --name TEST --filename=temp.file --rw=write --size=64g --io_size=128g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=600 Run status group 0 (all jobs): WRITE: bw=1021MiB/s (1070MB/s), 1021MiB/s-1021MiB/s (1070MB/s-1070MB/s), io=128GiB (137GB), run=128418-128418msec

astrodb commented 2 months ago

In these short tests, write IO on the NVMe Cinder volume is equivalent and better than that seen on a bare metal NVMe SSD.

Type dd test speed fio test speed
Cinder NVMe 634 MB/s 1298MB/s
Bare metal 438 MB/s 1070MB/s
Ceph SSD 510 MB/s 1596MB/s
astrodb commented 2 months ago

To add another comparison point, I've run the same tests on a Lasair VM using the CephSSD volume:

root@lasair-lsst-dev-cassandranodes-1:/var/lib/cassandra# dd if=/dev/zero of=local-test2 conv=fsync bs=512k count=2000 2000+0 records in 2000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.05627 s, 510 MB/s

root@lasair-lsst-dev-cassandranodes-1:/var/lib/cassandra# fio --name TEST --filename=temp.file --rw=write --size=64g --io_size=128g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=600 Run status group 0 (all jobs): WRITE: bw=1522MiB/s (1596MB/s), 1522MiB/s-1522MiB/s (1596MB/s-1596MB/s), io=128GiB (137GB), run=86110-86110msec

GregBlow commented 1 month ago

Another data point, ZFS raidz2 array of 14 nvme disks terms qservoneuclid

root@qservoneuclid:/qserv-data# dd if=/dev/zero of=local-test2 conv=fsync bs=512k count=2000
2000+0 records in
2000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.439601 s, 2.4 GB/s
root@qservoneuclid:/qserv-data# fio --name TEST --filename=temp.file --rw=write --size=64g --io_size=128g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=600
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
TEST: Laying out IO file (1 file / 65536MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=3905MiB/s][w=3905 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=512452: Thu Oct 10 15:12:33 2024
  write: IOPS=3607, BW=3608MiB/s (3783MB/s)(128GiB/36332msec); 0 zone resets
    slat (usec): min=169, max=4766, avg=250.53, stdev=123.66
    clat (usec): min=2, max=355928, avg=8591.34, stdev=14552.71
     lat (usec): min=263, max=356169, avg=8842.00, stdev=14565.04
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    7], 60.00th=[    9],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   15], 99.50th=[   17], 99.90th=[  300], 99.95th=[  330],
     | 99.99th=[  355]
   bw (  MiB/s): min= 1282, max= 4222, per=100.00%, avg=3610.14, stdev=688.63, samples=72
   iops        : min= 1282, max= 4222, avg=3610.12, stdev=688.62, samples=72
  lat (usec)   : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=75.90%, 20=23.75%, 50=0.01%
  lat (msec)   : 250=0.14%, 500=0.17%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=741, max=1252, avg=1012.08, stdev=137.27
    sync percentiles (nsec):
     |  1.00th=[  740],  5.00th=[  740], 10.00th=[  924], 20.00th=[  948],
     | 30.00th=[  964], 40.00th=[  972], 50.00th=[  988], 60.00th=[  988],
     | 70.00th=[ 1020], 80.00th=[ 1160], 90.00th=[ 1240], 95.00th=[ 1256],
     | 99.00th=[ 1256], 99.50th=[ 1256], 99.90th=[ 1256], 99.95th=[ 1256],
     | 99.99th=[ 1256]
  cpu          : usr=7.07%, sys=85.10%, ctx=2909, majf=0, minf=22
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,131072,0,13 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=3608MiB/s (3783MB/s), 3608MiB/s-3608MiB/s (3783MB/s-3783MB/s), io=128GiB (137GB), run=36332-36332msec
GregBlow commented 1 month ago

on qservoneuclid, running a small variation of tests from https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/

Single 4KiB random write process

root@qservoneuclid:/qserv-data# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.28
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=145MiB/s][w=37.1k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=1818403: Thu Oct 10 15:34:15 2024
  write: IOPS=37.5k, BW=147MiB/s (154MB/s)(8802MiB/60078msec); 0 zone resets
    slat (nsec): min=521, max=1222.8k, avg=987.98, stdev=1914.53
    clat (nsec): min=360, max=13935k, avg=25297.98, stdev=28976.46
     lat (usec): min=15, max=13936, avg=26.29, stdev=29.10
    clat percentiles (usec):
     |  1.00th=[   16],  5.00th=[   16], 10.00th=[   17], 20.00th=[   20],
     | 30.00th=[   21], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   25], 90.00th=[   36], 95.00th=[   47],
     | 99.00th=[  103], 99.50th=[  137], 99.90th=[  273], 99.95th=[  465],
     | 99.99th=[  947]
   bw (  KiB/s): min=112880, max=199176, per=100.00%, avg=150249.48, stdev=11130.97, samples=119
   iops        : min=28220, max=49794, avg=37562.35, stdev=2782.76, samples=119
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.01%, 20=22.08%, 50=73.58%, 100=3.27%, 250=0.96%
  lat (usec)   : 500=0.07%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=5.21%, sys=8.63%, ctx=2254228, majf=0, minf=741
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2253217,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=8802MiB (9229MB), run=60078-60078msec
root@qservoneuclid:/qserv-data# fio --name=random-write --ioengine=posixaio --rw=randrw --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=99.9MiB/s,w=99.3MiB/s][r=25.6k,w=25.4k IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2243566: Thu Oct 10 15:36:40 2024
  read: IOPS=25.4k, BW=99.1MiB/s (104MB/s)(5955MiB/60082msec)
    slat (nsec): min=511, max=1569.4k, avg=968.36, stdev=2286.70
    clat (nsec): min=191, max=2636.0k, avg=12800.65, stdev=17283.05
     lat (usec): min=10, max=2636, avg=13.77, stdev=17.47
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   13], 90.00th=[   14], 95.00th=[   16],
     | 99.00th=[   35], 99.50th=[   40], 99.90th=[  159], 99.95th=[  351],
     | 99.99th=[  807]
   bw (  KiB/s): min=83560, max=119296, per=100.00%, avg=101663.93, stdev=4852.39, samples=119
   iops        : min=20890, max=29824, avg=25415.98, stdev=1213.10, samples=119
  write: IOPS=25.3k, BW=99.0MiB/s (104MB/s)(5948MiB/60082msec); 0 zone resets
    slat (nsec): min=531, max=1188.7k, avg=978.08, stdev=1710.90
    clat (nsec): min=771, max=3178.2k, avg=23985.12, stdev=24241.94
     lat (usec): min=15, max=3182, avg=24.96, stdev=24.36
    clat percentiles (usec):
     |  1.00th=[   16],  5.00th=[   17], 10.00th=[   17], 20.00th=[   20],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   21], 80.00th=[   22], 90.00th=[   29], 95.00th=[   44],
     | 99.00th=[  100], 99.50th=[  133], 99.90th=[  249], 99.95th=[  441],
     | 99.99th=[  914]
   bw (  KiB/s): min=83496, max=118848, per=100.00%, avg=101522.15, stdev=4882.80, samples=119
   iops        : min=20874, max=29712, avg=25380.54, stdev=1220.68, samples=119
  lat (nsec)   : 250=0.01%, 1000=0.01%
  lat (usec)   : 10=0.82%, 20=65.49%, 50=31.77%, 100=1.35%, 250=0.48%
  lat (usec)   : 500=0.05%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=6.50%, sys=11.41%, ctx=3048296, majf=0, minf=557
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1524604,1522656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=99.1MiB/s (104MB/s), 99.1MiB/s-99.1MiB/s (104MB/s-104MB/s), io=5955MiB (6245MB), run=60082-60082msec
  WRITE: bw=99.0MiB/s (104MB/s), 99.0MiB/s-99.0MiB/s (104MB/s-104MB/s), io=5948MiB (6237MB), run=60082-60082msec

16 parallel 64KiB random write processes

root@qservoneuclid:/qserv-data# fio --name=random-write --ioengine=posixaio --rw=randrw --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.28
Starting 16 processes
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
Jobs: 16 (f=16): [F(16)][100.0%][r=1024KiB/s,w=1024KiB/s][r=16,w=16 IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2540778: Thu Oct 10 15:40:16 2024
  read: IOPS=4124, BW=258MiB/s (270MB/s)(15.6GiB/61830msec)
    slat (nsec): min=40, max=5025.0k, avg=255.50, stdev=12399.19
    clat (nsec): min=1562, max=76856k, avg=1786501.87, stdev=2722156.76
     lat (usec): min=19, max=76856, avg=1786.76, stdev=2722.17
    clat percentiles (usec):
     |  1.00th=[  429],  5.00th=[  611], 10.00th=[  725], 20.00th=[  889],
     | 30.00th=[ 1029], 40.00th=[ 1172], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3392],
     | 99.00th=[12911], 99.50th=[16909], 99.90th=[36963], 99.95th=[46400],
     | 99.99th=[67634]
   bw (  KiB/s): min=205568, max=343040, per=6.49%, avg=272467.72, stdev=27274.11, samples=119
   iops        : min= 3212, max= 5360, avg=4257.25, stdev=426.16, samples=119
  write: IOPS=4129, BW=258MiB/s (271MB/s)(15.6GiB/61830msec); 0 zone resets
    slat (nsec): min=140, max=5580.4k, avg=974.08, stdev=15921.08
    clat (usec): min=80, max=78459, avg=1968.00, stdev=2943.76
     lat (usec): min=83, max=78461, avg=1968.98, stdev=2943.87
    clat percentiles (usec):
     |  1.00th=[  502],  5.00th=[  693], 10.00th=[  824], 20.00th=[ 1004],
     | 30.00th=[ 1156], 40.00th=[ 1303], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2073], 90.00th=[ 2540], 95.00th=[ 3752],
     | 99.00th=[14091], 99.50th=[18220], 99.90th=[41157], 99.95th=[55313],
     | 99.99th=[70779]
   bw (  KiB/s): min=202624, max=357376, per=6.49%, avg=272665.76, stdev=27419.24, samples=119
   iops        : min= 3166, max= 5584, avg=4260.35, stdev=428.44, samples=119
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.04%, 500=1.47%, 750=7.56%, 1000=14.74%
  lat (msec)   : 2=56.75%, 4=14.91%, 10=2.53%, 20=1.60%, 50=0.35%
  lat (msec)   : 100=0.05%
  cpu          : usr=1.70%, sys=1.01%, ctx=293520, majf=0, minf=1787
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=3.2%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=255013,255304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540779: Thu Oct 10 15:40:16 2024
  read: IOPS=4105, BW=257MiB/s (269MB/s)(15.5GiB/61830msec)
    slat (nsec): min=40, max=1458.0k, avg=262.38, stdev=6442.49
    clat (nsec): min=651, max=90050k, avg=1787748.95, stdev=2871444.86
     lat (usec): min=20, max=90049, avg=1788.01, stdev=2871.48
    clat percentiles (usec):
     |  1.00th=[  396],  5.00th=[  586], 10.00th=[  701], 20.00th=[  873],
     | 30.00th=[ 1012], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1450],
     | 70.00th=[ 1631], 80.00th=[ 1876], 90.00th=[ 2311], 95.00th=[ 3359],
     | 99.00th=[13304], 99.50th=[17695], 99.90th=[41681], 99.95th=[49546],
     | 99.99th=[66847]
   bw (  KiB/s): min=207872, max=454656, per=6.46%, avg=271225.21, stdev=33814.28, samples=119
   iops        : min= 3248, max= 7104, avg=4237.85, stdev=528.39, samples=119
  write: IOPS=4122, BW=258MiB/s (270MB/s)(15.6GiB/61830msec); 0 zone resets
    slat (nsec): min=140, max=24441k, avg=1206.70, stdev=54136.93
    clat (usec): min=46, max=90046, avg=1976.34, stdev=3095.10
     lat (usec): min=91, max=90051, avg=1977.55, stdev=3097.43
    clat percentiles (usec):
     |  1.00th=[  449],  5.00th=[  660], 10.00th=[  799], 20.00th=[  979],
     | 30.00th=[ 1139], 40.00th=[ 1287], 50.00th=[ 1434], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2057], 90.00th=[ 2540], 95.00th=[ 3818],
     | 99.00th=[14484], 99.50th=[19006], 99.90th=[44827], 99.95th=[54789],
     | 99.99th=[71828]
   bw (  KiB/s): min=205952, max=472832, per=6.48%, avg=272322.29, stdev=33949.93, samples=119
   iops        : min= 3218, max= 7388, avg=4254.99, stdev=530.49, samples=119
  lat (nsec)   : 750=0.01%
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.06%, 500=2.11%
  lat (usec)   : 750=8.14%, 1000=14.75%
  lat (msec)   : 2=55.87%, 4=14.49%, 10=2.49%, 20=1.66%, 50=0.38%
  lat (msec)   : 100=0.05%
  cpu          : usr=1.77%, sys=0.96%, ctx=290333, majf=0, minf=1851
  IO depths    : 1=0.1%, 2=0.2%, 4=0.6%, 8=51.3%, 16=47.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.0%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=253820,254918,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540780: Thu Oct 10 15:40:16 2024
  read: IOPS=4149, BW=259MiB/s (272MB/s)(15.7GiB/61830msec)
    slat (nsec): min=40, max=1534.3k, avg=231.41, stdev=5244.98
    clat (usec): min=17, max=84865, avg=1780.98, stdev=2700.00
     lat (usec): min=18, max=84866, avg=1781.21, stdev=2700.01
    clat percentiles (usec):
     |  1.00th=[  392],  5.00th=[  578], 10.00th=[  709], 20.00th=[  881],
     | 30.00th=[ 1029], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3326],
     | 99.00th=[13173], 99.50th=[17433], 99.90th=[35914], 99.95th=[45351],
     | 99.99th=[65799]
   bw (  KiB/s): min=218059, max=425216, per=6.52%, avg=273712.42, stdev=31066.88, samples=119
   iops        : min= 3407, max= 6644, avg=4276.71, stdev=485.44, samples=119
  write: IOPS=4145, BW=259MiB/s (272MB/s)(15.6GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=3332.1k, avg=1003.19, stdev=14940.61
    clat (usec): min=4, max=85730, avg=1954.03, stdev=2865.09
     lat (usec): min=46, max=85731, avg=1955.03, stdev=2865.31
    clat percentiles (usec):
     |  1.00th=[  445],  5.00th=[  660], 10.00th=[  799], 20.00th=[  996],
     | 30.00th=[ 1139], 40.00th=[ 1303], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2073], 90.00th=[ 2540], 95.00th=[ 3621],
     | 99.00th=[13960], 99.50th=[18744], 99.90th=[41157], 99.95th=[46400],
     | 99.99th=[65799]
   bw (  KiB/s): min=219904, max=439040, per=6.50%, avg=273321.13, stdev=30983.72, samples=119
   iops        : min= 3436, max= 6860, avg=4270.59, stdev=484.14, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.03%
  lat (usec)   : 500=2.18%, 750=7.80%, 1000=14.31%
  lat (msec)   : 2=56.07%, 4=15.14%, 10=2.46%, 20=1.58%, 50=0.40%
  lat (msec)   : 100=0.03%
  cpu          : usr=1.68%, sys=1.07%, ctx=294657, majf=0, minf=1423
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.0%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=256555,256312,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540781: Thu Oct 10 15:40:16 2024
  read: IOPS=4140, BW=259MiB/s (271MB/s)(15.6GiB/61830msec)
    slat (nsec): min=40, max=2414.3k, avg=258.40, stdev=7190.92
    clat (usec): min=15, max=77047, avg=1777.44, stdev=2633.78
     lat (usec): min=17, max=77048, avg=1777.70, stdev=2633.80
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  586], 10.00th=[  709], 20.00th=[  881],
     | 30.00th=[ 1029], 40.00th=[ 1172], 50.00th=[ 1319], 60.00th=[ 1467],
     | 70.00th=[ 1663], 80.00th=[ 1909], 90.00th=[ 2343], 95.00th=[ 3261],
     | 99.00th=[13173], 99.50th=[16909], 99.90th=[35390], 99.95th=[45351],
     | 99.99th=[68682]
   bw (  KiB/s): min=206336, max=446720, per=6.51%, avg=273464.73, stdev=34301.18, samples=119
   iops        : min= 3224, max= 6980, avg=4272.83, stdev=535.98, samples=119
  write: IOPS=4152, BW=260MiB/s (272MB/s)(15.7GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=39494k, avg=1238.60, stdev=82911.87
    clat (usec): min=43, max=77116, avg=1958.42, stdev=2805.14
     lat (usec): min=147, max=77117, avg=1959.66, stdev=2808.47
    clat percentiles (usec):
     |  1.00th=[  453],  5.00th=[  668], 10.00th=[  807], 20.00th=[  988],
     | 30.00th=[ 1156], 40.00th=[ 1303], 50.00th=[ 1467], 60.00th=[ 1631],
     | 70.00th=[ 1827], 80.00th=[ 2089], 90.00th=[ 2540], 95.00th=[ 3621],
     | 99.00th=[13960], 99.50th=[17957], 99.90th=[37487], 99.95th=[46924],
     | 99.99th=[68682]
   bw (  KiB/s): min=213120, max=450816, per=6.52%, avg=274204.34, stdev=33138.03, samples=119
   iops        : min= 3330, max= 7044, avg=4284.40, stdev=517.79, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.04%, 500=2.05%
  lat (usec)   : 750=7.76%, 1000=14.38%
  lat (msec)   : 2=55.79%, 4=15.58%, 10=2.33%, 20=1.68%, 50=0.34%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.81%, sys=0.94%, ctx=296797, majf=0, minf=1619
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.2%, 16=48.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=256006,256738,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540782: Thu Oct 10 15:40:16 2024
  read: IOPS=4076, BW=255MiB/s (267MB/s)(15.4GiB/61830msec)
    slat (nsec): min=40, max=1055.1k, avg=236.98, stdev=5081.77
    clat (usec): min=17, max=76071, avg=1812.18, stdev=2911.28
     lat (usec): min=17, max=76071, avg=1812.42, stdev=2911.29
    clat percentiles (usec):
     |  1.00th=[  392],  5.00th=[  586], 10.00th=[  709], 20.00th=[  881],
     | 30.00th=[ 1029], 40.00th=[ 1172], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3523],
     | 99.00th=[13435], 99.50th=[17957], 99.90th=[42730], 99.95th=[54789],
     | 99.99th=[66847]
   bw (  KiB/s): min=196992, max=446464, per=6.41%, avg=269209.74, stdev=33977.52, samples=119
   iops        : min= 3078, max= 6976, avg=4206.35, stdev=530.93, samples=119
  write: IOPS=4087, BW=255MiB/s (268MB/s)(15.4GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=11967k, avg=1142.24, stdev=31934.15
    clat (usec): min=9, max=76071, avg=1982.30, stdev=3031.33
     lat (usec): min=98, max=76072, avg=1983.45, stdev=3031.82
    clat percentiles (usec):
     |  1.00th=[  445],  5.00th=[  668], 10.00th=[  807], 20.00th=[  996],
     | 30.00th=[ 1156], 40.00th=[ 1303], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2073], 90.00th=[ 2540], 95.00th=[ 3884],
     | 99.00th=[14222], 99.50th=[18744], 99.90th=[45876], 99.95th=[55837],
     | 99.99th=[67634]
   bw (  KiB/s): min=196224, max=439808, per=6.42%, avg=269892.36, stdev=32718.10, samples=119
   iops        : min= 3066, max= 6872, avg=4217.02, stdev=511.26, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.05%
  lat (usec)   : 500=2.15%, 750=7.63%, 1000=14.29%
  lat (msec)   : 2=56.31%, 4=14.87%, 10=2.61%, 20=1.66%, 50=0.36%
  lat (msec)   : 100=0.07%
  cpu          : usr=1.62%, sys=1.09%, ctx=291428, majf=0, minf=1626
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.2%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.0%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=252059,252735,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540783: Thu Oct 10 15:40:16 2024
  read: IOPS=4175, BW=261MiB/s (274MB/s)(15.8GiB/61830msec)
    slat (nsec): min=40, max=1257.4k, avg=247.11, stdev=6039.95
    clat (usec): min=4, max=79705, avg=1766.24, stdev=2613.11
     lat (usec): min=21, max=79705, avg=1766.48, stdev=2613.12
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  578], 10.00th=[  701], 20.00th=[  873],
     | 30.00th=[ 1020], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3392],
     | 99.00th=[12911], 99.50th=[16450], 99.90th=[35390], 99.95th=[43779],
     | 99.99th=[64750]
   bw (  KiB/s): min=214784, max=496640, per=6.57%, avg=275830.44, stdev=34742.30, samples=119
   iops        : min= 3356, max= 7760, avg=4309.81, stdev=542.87, samples=119
  write: IOPS=4168, BW=261MiB/s (273MB/s)(15.7GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=3983.2k, avg=1035.73, stdev=18531.86
    clat (usec): min=17, max=80472, avg=1946.87, stdev=2811.54
     lat (usec): min=56, max=80473, avg=1947.90, stdev=2811.62
    clat percentiles (usec):
     |  1.00th=[  433],  5.00th=[  668], 10.00th=[  799], 20.00th=[  988],
     | 30.00th=[ 1139], 40.00th=[ 1287], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2073], 90.00th=[ 2540], 95.00th=[ 3752],
     | 99.00th=[13960], 99.50th=[17433], 99.90th=[38536], 99.95th=[50070],
     | 99.99th=[67634]
   bw (  KiB/s): min=210432, max=483968, per=6.55%, avg=275264.49, stdev=34200.60, samples=119
   iops        : min= 3288, max= 7562, avg=4300.97, stdev=534.39, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.05%
  lat (usec)   : 500=2.32%, 750=7.76%, 1000=14.78%
  lat (msec)   : 2=55.66%, 4=14.89%, 10=2.57%, 20=1.63%, 50=0.31%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.81%, sys=0.97%, ctx=298600, majf=0, minf=1947
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.0%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=258186,257717,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540784: Thu Oct 10 15:40:16 2024
  read: IOPS=4133, BW=258MiB/s (271MB/s)(15.6GiB/61830msec)
    slat (nsec): min=40, max=1931.5k, avg=264.15, stdev=8246.33
    clat (usec): min=18, max=92957, avg=1784.62, stdev=2736.46
     lat (usec): min=20, max=92958, avg=1784.88, stdev=2736.59
    clat percentiles (usec):
     |  1.00th=[  388],  5.00th=[  578], 10.00th=[  709], 20.00th=[  881],
     | 30.00th=[ 1029], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3359],
     | 99.00th=[13173], 99.50th=[17433], 99.90th=[37487], 99.95th=[45876],
     | 99.99th=[68682]
   bw (  KiB/s): min=185216, max=469632, per=6.49%, avg=272538.18, stdev=31593.16, samples=119
   iops        : min= 2894, max= 7338, avg=4258.38, stdev=493.66, samples=119
  write: IOPS=4135, BW=258MiB/s (271MB/s)(15.6GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=3610.7k, avg=1061.70, stdev=14831.67
    clat (usec): min=10, max=92956, avg=1961.73, stdev=2840.88
     lat (usec): min=144, max=92957, avg=1962.79, stdev=2840.92
    clat percentiles (usec):
     |  1.00th=[  441],  5.00th=[  668], 10.00th=[  807], 20.00th=[  996],
     | 30.00th=[ 1156], 40.00th=[ 1303], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2057], 90.00th=[ 2540], 95.00th=[ 3785],
     | 99.00th=[14222], 99.50th=[18482], 99.90th=[38536], 99.95th=[46924],
     | 99.99th=[65799]
   bw (  KiB/s): min=196608, max=470784, per=6.49%, avg=272755.59, stdev=30778.60, samples=119
   iops        : min= 3072, max= 7356, avg=4261.75, stdev=480.93, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.04%, 500=2.20%
  lat (usec)   : 750=7.69%, 1000=14.12%
  lat (msec)   : 2=56.64%, 4=14.74%, 10=2.54%, 20=1.63%, 50=0.36%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.68%, sys=1.06%, ctx=293606, majf=0, minf=1533
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=47.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=255571,255703,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540785: Thu Oct 10 15:40:16 2024
  read: IOPS=4128, BW=258MiB/s (271MB/s)(15.6GiB/61824msec)
    slat (nsec): min=40, max=1450.8k, avg=239.44, stdev=5721.27
    clat (usec): min=16, max=98861, avg=1784.45, stdev=2619.26
     lat (usec): min=17, max=98861, avg=1784.69, stdev=2619.35
    clat percentiles (usec):
     |  1.00th=[  412],  5.00th=[  603], 10.00th=[  734], 20.00th=[  906],
     | 30.00th=[ 1057], 40.00th=[ 1188], 50.00th=[ 1336], 60.00th=[ 1483],
     | 70.00th=[ 1680], 80.00th=[ 1909], 90.00th=[ 2343], 95.00th=[ 3294],
     | 99.00th=[12911], 99.50th=[16188], 99.90th=[33817], 99.95th=[45351],
     | 99.99th=[63701]
   bw (  KiB/s): min=201344, max=430208, per=6.48%, avg=272150.95, stdev=34284.59, samples=119
   iops        : min= 3146, max= 6722, avg=4252.32, stdev=535.70, samples=119
  write: IOPS=4131, BW=258MiB/s (271MB/s)(15.6GiB/61824msec); 0 zone resets
    slat (nsec): min=150, max=8548.3k, avg=1017.91, stdev=21704.40
    clat (usec): min=42, max=98860, avg=1966.34, stdev=2739.57
     lat (usec): min=45, max=98861, avg=1967.36, stdev=2739.74
    clat percentiles (usec):
     |  1.00th=[  461],  5.00th=[  685], 10.00th=[  824], 20.00th=[ 1012],
     | 30.00th=[ 1172], 40.00th=[ 1319], 50.00th=[ 1483], 60.00th=[ 1647],
     | 70.00th=[ 1844], 80.00th=[ 2114], 90.00th=[ 2573], 95.00th=[ 3785],
     | 99.00th=[13960], 99.50th=[17171], 99.90th=[35914], 99.95th=[45351],
     | 99.99th=[62653]
   bw (  KiB/s): min=202752, max=442880, per=6.48%, avg=272372.97, stdev=34607.10, samples=119
   iops        : min= 3168, max= 6920, avg=4255.78, stdev=540.75, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.03%, 500=1.84%
  lat (usec)   : 750=7.12%, 1000=13.94%
  lat (msec)   : 2=56.68%, 4=15.88%, 10=2.59%, 20=1.56%, 50=0.32%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.78%, sys=0.96%, ctx=295116, majf=0, minf=1549
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.0%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=255250,255444,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540786: Thu Oct 10 15:40:16 2024
  read: IOPS=4176, BW=261MiB/s (274MB/s)(15.8GiB/61830msec)
    slat (nsec): min=40, max=1114.4k, avg=230.09, stdev=4803.03
    clat (usec): min=6, max=79713, avg=1755.98, stdev=2647.16
     lat (usec): min=22, max=79713, avg=1756.21, stdev=2647.16
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  594], 10.00th=[  717], 20.00th=[  881],
     | 30.00th=[ 1029], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1876], 90.00th=[ 2311], 95.00th=[ 3163],
     | 99.00th=[12911], 99.50th=[16450], 99.90th=[39584], 99.95th=[46400],
     | 99.99th=[62653]
   bw (  KiB/s): min=196608, max=484480, per=6.56%, avg=275736.12, stdev=34684.88, samples=119
   iops        : min= 3072, max= 7570, avg=4308.35, stdev=541.96, samples=119
  write: IOPS=4190, BW=262MiB/s (275MB/s)(15.8GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=10832k, avg=1011.00, stdev=24715.74
    clat (usec): min=6, max=71449, avg=1946.73, stdev=2842.55
     lat (usec): min=80, max=71453, avg=1947.74, stdev=2842.69
    clat percentiles (usec):
     |  1.00th=[  445],  5.00th=[  668], 10.00th=[  807], 20.00th=[  996],
     | 30.00th=[ 1139], 40.00th=[ 1287], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2073], 90.00th=[ 2540], 95.00th=[ 3589],
     | 99.00th=[14091], 99.50th=[17695], 99.90th=[40109], 99.95th=[51643],
     | 99.99th=[65274]
   bw (  KiB/s): min=210688, max=498816, per=6.59%, avg=276846.19, stdev=34952.52, samples=119
   iops        : min= 3292, max= 7794, avg=4325.68, stdev=546.14, samples=119
  lat (usec)   : 10=0.01%, 50=0.01%, 100=0.01%, 250=0.04%, 500=2.09%
  lat (usec)   : 750=7.68%, 1000=14.45%
  lat (msec)   : 2=56.43%, 4=15.02%, 10=2.33%, 20=1.59%, 50=0.31%
  lat (msec)   : 100=0.05%
  cpu          : usr=1.75%, sys=1.00%, ctx=299734, majf=0, minf=1874
  IO depths    : 1=0.1%, 2=0.2%, 4=0.5%, 8=51.2%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=258224,259117,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540787: Thu Oct 10 15:40:16 2024
  read: IOPS=4130, BW=258MiB/s (271MB/s)(15.6GiB/61830msec)
    slat (nsec): min=40, max=2424.1k, avg=256.26, stdev=8830.90
    clat (usec): min=18, max=83127, avg=1784.09, stdev=2746.63
     lat (usec): min=19, max=83128, avg=1784.35, stdev=2746.68
    clat percentiles (usec):
     |  1.00th=[  396],  5.00th=[  586], 10.00th=[  709], 20.00th=[  881],
     | 30.00th=[ 1020], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3425],
     | 99.00th=[13042], 99.50th=[16909], 99.90th=[38536], 99.95th=[47449],
     | 99.99th=[65799]
   bw (  KiB/s): min=192640, max=469888, per=6.49%, avg=272637.24, stdev=31303.37, samples=119
   iops        : min= 3010, max= 7342, avg=4259.90, stdev=489.13, samples=119
  write: IOPS=4118, BW=257MiB/s (270MB/s)(15.5GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=2212.5k, avg=950.73, stdev=11705.59
    clat (usec): min=67, max=83126, avg=1971.94, stdev=2940.13
     lat (usec): min=91, max=83127, avg=1972.89, stdev=2940.33
    clat percentiles (usec):
     |  1.00th=[  453],  5.00th=[  668], 10.00th=[  807], 20.00th=[  996],
     | 30.00th=[ 1139], 40.00th=[ 1303], 50.00th=[ 1450], 60.00th=[ 1614],
     | 70.00th=[ 1811], 80.00th=[ 2073], 90.00th=[ 2573], 95.00th=[ 3851],
     | 99.00th=[14091], 99.50th=[18482], 99.90th=[42206], 99.95th=[49546],
     | 99.99th=[66323]
   bw (  KiB/s): min=197760, max=455936, per=6.47%, avg=271791.12, stdev=29951.58, samples=119
   iops        : min= 3090, max= 7124, avg=4246.68, stdev=468.01, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.06%, 500=2.02%
  lat (usec)   : 750=7.72%, 1000=14.60%
  lat (msec)   : 2=56.11%, 4=14.84%, 10=2.66%, 20=1.56%, 50=0.38%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.75%, sys=0.98%, ctx=292782, majf=0, minf=1600
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=2.9%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=255405,254661,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540788: Thu Oct 10 15:40:16 2024
  read: IOPS=4015, BW=251MiB/s (263MB/s)(15.2GiB/61830msec)
    slat (nsec): min=40, max=1625.7k, avg=260.89, stdev=7475.61
    clat (usec): min=7, max=70833, avg=1834.63, stdev=2708.57
     lat (usec): min=14, max=70833, avg=1834.89, stdev=2708.58
    clat percentiles (usec):
     |  1.00th=[  429],  5.00th=[  635], 10.00th=[  766], 20.00th=[  938],
     | 30.00th=[ 1090], 40.00th=[ 1221], 50.00th=[ 1369], 60.00th=[ 1516],
     | 70.00th=[ 1696], 80.00th=[ 1942], 90.00th=[ 2376], 95.00th=[ 3359],
     | 99.00th=[13173], 99.50th=[17171], 99.90th=[39060], 99.95th=[49021],
     | 99.99th=[63177]
   bw (  KiB/s): min=209152, max=424704, per=6.30%, avg=264809.68, stdev=26903.96, samples=119
   iops        : min= 3268, max= 6636, avg=4137.61, stdev=420.38, samples=119
  write: IOPS=4020, BW=251MiB/s (263MB/s)(15.2GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=6834.5k, avg=1025.01, stdev=17897.42
    clat (usec): min=64, max=70992, avg=2020.43, stdev=2879.31
     lat (usec): min=104, max=70994, avg=2021.45, stdev=2879.39
    clat percentiles (usec):
     |  1.00th=[  490],  5.00th=[  725], 10.00th=[  865], 20.00th=[ 1045],
     | 30.00th=[ 1205], 40.00th=[ 1352], 50.00th=[ 1500], 60.00th=[ 1680],
     | 70.00th=[ 1860], 80.00th=[ 2114], 90.00th=[ 2606], 95.00th=[ 3785],
     | 99.00th=[14222], 99.50th=[18744], 99.90th=[41157], 99.95th=[50070],
     | 99.99th=[66323]
   bw (  KiB/s): min=214400, max=420736, per=6.30%, avg=265000.80, stdev=26749.31, samples=119
   iops        : min= 3350, max= 6574, avg=4140.61, stdev=417.97, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.03%
  lat (usec)   : 500=1.46%, 750=6.04%, 1000=13.12%
  lat (msec)   : 2=58.18%, 4=16.61%, 10=2.50%, 20=1.64%, 50=0.38%
  lat (msec)   : 100=0.05%
  cpu          : usr=1.67%, sys=1.04%, ctx=287755, majf=0, minf=1536
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.2%, 16=48.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=248269,248557,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540789: Thu Oct 10 15:40:16 2024
  read: IOPS=4003, BW=250MiB/s (262MB/s)(15.1GiB/61830msec)
    slat (nsec): min=40, max=1460.1k, avg=258.61, stdev=5747.04
    clat (usec): min=17, max=76746, avg=1841.00, stdev=2694.14
     lat (usec): min=19, max=76746, avg=1841.26, stdev=2694.16
    clat percentiles (usec):
     |  1.00th=[  441],  5.00th=[  652], 10.00th=[  783], 20.00th=[  955],
     | 30.00th=[ 1106], 40.00th=[ 1237], 50.00th=[ 1385], 60.00th=[ 1532],
     | 70.00th=[ 1713], 80.00th=[ 1958], 90.00th=[ 2409], 95.00th=[ 3392],
     | 99.00th=[12911], 99.50th=[16319], 99.90th=[37487], 99.95th=[51643],
     | 99.99th=[69731]
   bw (  KiB/s): min=200704, max=447360, per=6.30%, avg=264498.22, stdev=31254.18, samples=119
   iops        : min= 3136, max= 6990, avg=4132.74, stdev=488.35, samples=119
  write: IOPS=4014, BW=251MiB/s (263MB/s)(15.1GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=3159.6k, avg=1039.09, stdev=15250.19
    clat (usec): min=5, max=76746, avg=2023.59, stdev=2865.45
     lat (usec): min=112, max=76747, avg=2024.63, stdev=2865.58
    clat percentiles (usec):
     |  1.00th=[  494],  5.00th=[  742], 10.00th=[  881], 20.00th=[ 1074],
     | 30.00th=[ 1221], 40.00th=[ 1369], 50.00th=[ 1532], 60.00th=[ 1696],
     | 70.00th=[ 1876], 80.00th=[ 2147], 90.00th=[ 2606], 95.00th=[ 3785],
     | 99.00th=[13829], 99.50th=[17433], 99.90th=[40633], 99.95th=[54264],
     | 99.99th=[69731]
   bw (  KiB/s): min=211840, max=456960, per=6.31%, avg=265179.34, stdev=30474.03, samples=119
   iops        : min= 3310, max= 7140, avg=4143.39, stdev=476.16, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 250=0.02%, 500=1.34%
  lat (usec)   : 750=5.52%, 1000=12.65%
  lat (msec)   : 2=58.71%, 4=17.22%, 10=2.58%, 20=1.61%, 50=0.30%
  lat (msec)   : 100=0.06%
  cpu          : usr=1.77%, sys=0.95%, ctx=288427, majf=0, minf=1606
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.2%, 16=48.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=247519,248212,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540790: Thu Oct 10 15:40:16 2024
  read: IOPS=4031, BW=252MiB/s (264MB/s)(15.2GiB/61830msec)
    slat (nsec): min=40, max=1048.3k, avg=228.78, stdev=4607.02
    clat (nsec): min=731, max=79183k, avg=1826587.71, stdev=2692323.67
     lat (usec): min=22, max=79182, avg=1826.82, stdev=2692.47
    clat percentiles (usec):
     |  1.00th=[  424],  5.00th=[  635], 10.00th=[  766], 20.00th=[  947],
     | 30.00th=[ 1090], 40.00th=[ 1221], 50.00th=[ 1369], 60.00th=[ 1532],
     | 70.00th=[ 1713], 80.00th=[ 1942], 90.00th=[ 2376], 95.00th=[ 3326],
     | 99.00th=[12780], 99.50th=[15926], 99.90th=[38011], 99.95th=[46924],
     | 99.99th=[74974]
   bw (  KiB/s): min=195840, max=441088, per=6.33%, avg=265823.00, stdev=31548.70, samples=119
   iops        : min= 3060, max= 6892, avg=4153.43, stdev=492.93, samples=119
  write: IOPS=4026, BW=252MiB/s (264MB/s)(15.2GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=4954.0k, avg=1089.91, stdev=19521.05
    clat (usec): min=5, max=79227, avg=2018.54, stdev=2946.66
     lat (usec): min=64, max=79227, avg=2019.63, stdev=2946.70
    clat percentiles (usec):
     |  1.00th=[  478],  5.00th=[  725], 10.00th=[  865], 20.00th=[ 1057],
     | 30.00th=[ 1205], 40.00th=[ 1369], 50.00th=[ 1516], 60.00th=[ 1680],
     | 70.00th=[ 1876], 80.00th=[ 2114], 90.00th=[ 2606], 95.00th=[ 3720],
     | 99.00th=[13960], 99.50th=[17957], 99.90th=[42730], 99.95th=[49546],
     | 99.99th=[73925]
   bw (  KiB/s): min=200704, max=428160, per=6.32%, avg=265487.93, stdev=30588.32, samples=119
   iops        : min= 3136, max= 6690, avg=4148.20, stdev=477.95, samples=119
  lat (nsec)   : 750=0.01%
  lat (usec)   : 10=0.01%, 50=0.01%, 100=0.01%, 250=0.03%, 500=1.54%
  lat (usec)   : 750=5.94%, 1000=12.89%
  lat (msec)   : 2=58.19%, 4=16.93%, 10=2.54%, 20=1.57%, 50=0.32%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.56%, sys=1.12%, ctx=288607, majf=0, minf=1772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.4%, 8=51.2%, 16=48.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.0%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=249291,248943,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540791: Thu Oct 10 15:40:16 2024
  read: IOPS=4132, BW=258MiB/s (271MB/s)(15.6GiB/61830msec)
    slat (nsec): min=40, max=1459.3k, avg=244.24, stdev=6418.29
    clat (nsec): min=942, max=74255k, avg=1776383.93, stdev=2684725.21
     lat (usec): min=19, max=74255, avg=1776.63, stdev=2684.77
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  570], 10.00th=[  701], 20.00th=[  873],
     | 30.00th=[ 1020], 40.00th=[ 1156], 50.00th=[ 1303], 60.00th=[ 1467],
     | 70.00th=[ 1647], 80.00th=[ 1893], 90.00th=[ 2343], 95.00th=[ 3359],
     | 99.00th=[12911], 99.50th=[16712], 99.90th=[38011], 99.95th=[45876],
     | 99.99th=[67634]
   bw (  KiB/s): min=196480, max=462336, per=6.50%, avg=272873.18, stdev=36997.33, samples=119
   iops        : min= 3070, max= 7224, avg=4263.60, stdev=578.09, samples=119
  write: IOPS=4154, BW=260MiB/s (272MB/s)(15.7GiB/61830msec); 0 zone resets
    slat (nsec): min=140, max=3221.9k, avg=999.54, stdev=14753.33
    clat (usec): min=67, max=74984, avg=1961.12, stdev=2849.25
     lat (usec): min=68, max=74985, avg=1962.12, stdev=2849.35
    clat percentiles (usec):
     |  1.00th=[  433],  5.00th=[  652], 10.00th=[  799], 20.00th=[  988],
     | 30.00th=[ 1139], 40.00th=[ 1287], 50.00th=[ 1450], 60.00th=[ 1631],
     | 70.00th=[ 1827], 80.00th=[ 2073], 90.00th=[ 2540], 95.00th=[ 3720],
     | 99.00th=[13960], 99.50th=[17957], 99.90th=[41157], 99.95th=[46400],
     | 99.99th=[67634]
   bw (  KiB/s): min=193280, max=470400, per=6.52%, avg=274298.45, stdev=36697.95, samples=119
   iops        : min= 3020, max= 7350, avg=4285.87, stdev=573.42, samples=119
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.04%, 500=2.41%
  lat (usec)   : 750=7.90%, 1000=14.52%
  lat (msec)   : 2=55.35%, 4=15.26%, 10=2.49%, 20=1.67%, 50=0.33%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.73%, sys=1.02%, ctx=294059, majf=0, minf=1737
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=255537,256896,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540792: Thu Oct 10 15:40:16 2024
  read: IOPS=4008, BW=251MiB/s (263MB/s)(15.1GiB/61830msec)
    slat (nsec): min=40, max=1660.3k, avg=257.64, stdev=7360.46
    clat (usec): min=18, max=80838, avg=1845.07, stdev=2685.84
     lat (usec): min=20, max=80838, avg=1845.33, stdev=2685.85
    clat percentiles (usec):
     |  1.00th=[  429],  5.00th=[  644], 10.00th=[  775], 20.00th=[  947],
     | 30.00th=[ 1090], 40.00th=[ 1237], 50.00th=[ 1369], 60.00th=[ 1532],
     | 70.00th=[ 1713], 80.00th=[ 1958], 90.00th=[ 2409], 95.00th=[ 3392],
     | 99.00th=[13042], 99.50th=[16712], 99.90th=[38011], 99.95th=[43254],
     | 99.99th=[64750]
   bw (  KiB/s): min=186624, max=432640, per=6.30%, avg=264695.76, stdev=31224.54, samples=119
   iops        : min= 2916, max= 6760, avg=4135.84, stdev=487.87, samples=119
  write: IOPS=4005, BW=250MiB/s (262MB/s)(15.1GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=4436.7k, avg=1084.21, stdev=20259.93
    clat (usec): min=5, max=80839, avg=2021.71, stdev=2806.88
     lat (usec): min=38, max=80840, avg=2022.79, stdev=2807.10
    clat percentiles (usec):
     |  1.00th=[  494],  5.00th=[  734], 10.00th=[  873], 20.00th=[ 1057],
     | 30.00th=[ 1205], 40.00th=[ 1369], 50.00th=[ 1516], 60.00th=[ 1680],
     | 70.00th=[ 1876], 80.00th=[ 2147], 90.00th=[ 2606], 95.00th=[ 3785],
     | 99.00th=[13960], 99.50th=[17433], 99.90th=[38536], 99.95th=[44827],
     | 99.99th=[62129]
   bw (  KiB/s): min=194560, max=434176, per=6.29%, avg=264513.87, stdev=30869.38, samples=119
   iops        : min= 3040, max= 6784, avg=4132.97, stdev=482.31, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.04%
  lat (usec)   : 500=1.39%, 750=5.80%, 1000=12.87%
  lat (msec)   : 2=58.21%, 4=17.10%, 10=2.47%, 20=1.75%, 50=0.32%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.61%, sys=1.12%, ctx=290857, majf=0, minf=1773
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.2%, 16=48.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=247839,247651,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=2540793: Thu Oct 10 15:40:16 2024
  read: IOPS=4096, BW=256MiB/s (268MB/s)(15.5GiB/61830msec)
    slat (nsec): min=40, max=1026.4k, avg=224.58, stdev=4181.15
    clat (usec): min=12, max=93256, avg=1804.96, stdev=2667.38
     lat (usec): min=12, max=93256, avg=1805.18, stdev=2667.38
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  611], 10.00th=[  734], 20.00th=[  914],
     | 30.00th=[ 1057], 40.00th=[ 1188], 50.00th=[ 1336], 60.00th=[ 1500],
     | 70.00th=[ 1680], 80.00th=[ 1926], 90.00th=[ 2376], 95.00th=[ 3425],
     | 99.00th=[13042], 99.50th=[16188], 99.90th=[37487], 99.95th=[46400],
     | 99.99th=[65274]
   bw (  KiB/s): min=204672, max=432768, per=6.43%, avg=270132.34, stdev=33205.28, samples=119
   iops        : min= 3198, max= 6762, avg=4220.80, stdev=518.84, samples=119
  write: IOPS=4084, BW=255MiB/s (268MB/s)(15.4GiB/61830msec); 0 zone resets
    slat (nsec): min=150, max=1675.8k, avg=958.91, stdev=10657.51
    clat (usec): min=47, max=93586, avg=1983.00, stdev=2817.58
     lat (usec): min=63, max=93587, avg=1983.96, stdev=2817.66
    clat percentiles (usec):
     |  1.00th=[  465],  5.00th=[  693], 10.00th=[  832], 20.00th=[ 1020],
     | 30.00th=[ 1172], 40.00th=[ 1319], 50.00th=[ 1483], 60.00th=[ 1647],
     | 70.00th=[ 1844], 80.00th=[ 2114], 90.00th=[ 2573], 95.00th=[ 3884],
     | 99.00th=[13829], 99.50th=[16909], 99.90th=[40109], 99.95th=[49021],
     | 99.99th=[67634]
   bw (  KiB/s): min=212096, max=437248, per=6.41%, avg=269387.17, stdev=31211.75, samples=119
   iops        : min= 3314, max= 6832, avg=4209.13, stdev=487.69, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.05%, 500=1.74%
  lat (usec)   : 750=7.07%, 1000=13.77%
  lat (msec)   : 2=56.88%, 4=15.81%, 10=2.64%, 20=1.67%, 50=0.31%
  lat (msec)   : 100=0.04%
  cpu          : usr=1.76%, sys=0.97%, ctx=291699, majf=0, minf=1633
  IO depths    : 1=0.1%, 2=0.1%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.1%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=253265,252537,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4102MiB/s (4301MB/s), 250MiB/s-261MiB/s (262MB/s-274MB/s), io=248GiB (266GB), run=61824-61830msec
  WRITE: bw=4105MiB/s (4305MB/s), 250MiB/s-262MiB/s (262MB/s-275MB/s), io=248GiB (266GB), run=61824-61830msec

Single 1MiB random write process

root@qservoneuclid:/qserv-data# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=1
fio-3.28
Starting 1 process
random-write: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=3383MiB/s][w=3383 IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2723260: Thu Oct 10 15:43:36 2024
  write: IOPS=3270, BW=3270MiB/s (3429MB/s)(193GiB/60429msec); 0 zone resets
    slat (usec): min=2, max=12101, avg= 8.58, stdev=30.53
    clat (nsec): min=521, max=14459k, avg=294361.93, stdev=169661.57
     lat (usec): min=164, max=14467, avg=302.95, stdev=174.77
    clat percentiles (usec):
     |  1.00th=[  206],  5.00th=[  215], 10.00th=[  219], 20.00th=[  225],
     | 30.00th=[  231], 40.00th=[  237], 50.00th=[  243], 60.00th=[  253],
     | 70.00th=[  281], 80.00th=[  318], 90.00th=[  412], 95.00th=[  545],
     | 99.00th=[  938], 99.50th=[ 1172], 99.90th=[ 1876], 99.95th=[ 2376],
     | 99.99th=[ 4752]
   bw (  MiB/s): min= 3182, max= 3454, per=100.00%, avg=3294.26, stdev=52.10, samples=119
   iops        : min= 3182, max= 3454, avg=3294.25, stdev=52.10, samples=119
  lat (nsec)   : 750=0.01%
  lat (usec)   : 250=58.22%, 500=35.54%, 750=4.19%, 1000=1.23%
  lat (msec)   : 2=0.73%, 4=0.07%, 10=0.01%, 20=0.01%
  cpu          : usr=3.19%, sys=0.80%, ctx=200560, majf=0, minf=6136
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,197609,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3270MiB/s (3429MB/s), 3270MiB/s-3270MiB/s (3429MB/s-3429MB/s), io=193GiB (207GB), run=60429-60429msec