fio

[wanghenshui]

fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=512 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest

mytest: (g=0): rw=write, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread mytest: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=287KiB/s][r=0,w=575 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=107146: Tue Apr 23 06:47:18 2024 write: IOPS=527, BW=264KiB/s (270kB/s)(30.9MiB/120002msec) slat (usec): min=422, max=20786, avg=1891.05, stdev=1311.53 clat (usec): min=599, max=439886, avg=187584.81, stdev=40509.57 lat (msec): min=2, max=442, avg=189.48, stdev=40.83 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 259], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 405], | 99.99th=[ 430] bw ( KiB/s): min= 123, max= 367, per=100.00%, avg=263.15, stdev=44.63, samples=240 iops : min= 246, max= 734, avg=526.31, stdev=89.25, samples=240 lat (usec) : 750=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%, 100=0.06% lat (msec) : 250=93.37%, 500=6.52% cpu : usr=0.48%, sys=2.14%, ctx=126512, majf=0, minf=6 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,63266,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=264KiB/s (270kB/s), 264KiB/s-264KiB/s (270kB/s-270kB/s), io=30.9MiB (32.4MB), run=120002-120002msec

Disk stats (read/write): vdb: ios=581/126787, merge=0/11, ticks=1909/117594, in_queue=119465, util=97.49%

fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=1024 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=559KiB/s][r=0,w=559 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=107149: Tue Apr 23 06:49:58 2024 write: IOPS=553, BW=554KiB/s (567kB/s)(64.9MiB/120002msec) slat (usec): min=366, max=33698, avg=1800.30, stdev=1254.34 clat (usec): min=819, max=561740, avg=178611.69, stdev=50078.99 lat (msec): min=2, max=569, avg=180.41, stdev=50.51 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 132], 10.00th=[ 138], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 251], | 99.00th=[ 422], 99.50th=[ 472], 99.90th=[ 514], 99.95th=[ 527], | 99.99th=[ 550] bw ( KiB/s): min= 186, max= 724, per=100.00%, avg=552.98, stdev=106.84, samples=240 iops : min= 186, max= 724, avg=552.98, stdev=106.84, samples=240 lat (usec) : 1000=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%, 100=0.03% lat (msec) : 250=94.87%, 500=4.87%, 750=0.17% cpu : usr=0.44%, sys=2.29%, ctx=132877, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,66462,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=554KiB/s (567kB/s), 554KiB/s-554KiB/s (567kB/s-567kB/s), io=64.9MiB (68.1MB), run=120002-120002msec

[wanghenshui]

sudo fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=1024 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=1537KiB/s][r=0,w=1537 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=5748: Tue Apr 23 15:01:34 2024 write: IOPS=2059, BW=2060KiB/s (2109kB/s)(241MiB/120001msec) slat (nsec): min=1083, max=39130k, avg=484885.60, stdev=1765074.96 clat (usec): min=2, max=135761, avg=48068.65, stdev=17575.64 lat (msec): min=2, max=135, avg=48.55, stdev=17.67 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 41], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 62], | 70.00th=[ 64], 80.00th=[ 65], 90.00th=[ 65], 95.00th=[ 68], | 99.00th=[ 78], 99.50th=[ 84], 99.90th=[ 99], 99.95th=[ 104], | 99.99th=[ 111] bw ( KiB/s): min= 1432, max= 8128, per=100.00%, avg=2060.80, stdev=737.99, samples=239 iops : min= 1432, max= 8128, avg=2060.80, stdev=737.99, samples=239 lat (usec) : 4=0.01%, 750=0.03%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.08%, 10=0.64%, 20=2.06%, 50=48.71% lat (msec) : 100=48.37%, 250=0.08% cpu : usr=0.23%, sys=1.45%, ctx=128527, majf=0, minf=1 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,247145,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=2060KiB/s (2109kB/s), 2060KiB/s-2060KiB/s (2109kB/s-2109kB/s), io=241MiB (253MB), run=120001-120001msec

Disk stats (read/write): nvme1n1: ios=0/341265, merge=0/29, ticks=0/1513051, in_queue=1513051, util=99.88%

[wanghenshui]

sudo fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=512 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread mytest: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=1092KiB/s][r=0,w=2184 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=48004: Tue Apr 23 14:51:22 2024 write: IOPS=2199, BW=1100KiB/s (1126kB/s)(129MiB/120002msec) slat (nsec): min=1098, max=37134k, avg=453989.62, stdev=1231372.05 clat (nsec): min=1931, max=127568k, avg=45003614.11, stdev=4110685.70 lat (msec): min=3, max=127, avg=45.46, stdev= 4.16 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 48], 95.00th=[ 48], | 99.00th=[ 55], 99.50th=[ 62], 99.90th=[ 77], 99.95th=[ 85], | 99.99th=[ 124] bw ( KiB/s): min= 976, max= 1576, per=100.00%, avg=1099.25, stdev=61.68, samples=240 iops : min= 1952, max= 3152, avg=2198.55, stdev=123.35, samples=240 lat (usec) : 2=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=98.10%, 100=1.85% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=1.33%, ctx=132320, majf=0, minf=1 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,263953,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=1100KiB/s (1126kB/s), 1100KiB/s-1100KiB/s (1126kB/s-1126kB/s), io=129MiB (135MB), run=120002-120002msec

Disk stats (read/write): nvme1n1: ios=0/362692, merge=0/15, ticks=0/388864, in_queue=388863, util=99.95%

[wanghenshui]

sudo fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=2048 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 2048B-2048B, (W) 2048B-2048B, (T) 2048B-2048B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=1565KiB/s][r=0,w=782 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=11690: Tue Apr 23 15:08:17 2024 write: IOPS=1530, BW=3061KiB/s (3134kB/s)(359MiB/120001msec) slat (nsec): min=1087, max=43971k, avg=652767.92, stdev=1969238.25 clat (nsec): min=1723, max=277006k, avg=64693534.48, stdev=44938153.28 lat (usec): min=1998, max=277007, avg=65346.39, stdev=45353.05 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 29], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 124], 80.00th=[ 126], 90.00th=[ 128], 95.00th=[ 130], | 99.00th=[ 140], 99.50th=[ 148], 99.90th=[ 209], 99.95th=[ 215], | 99.99th=[ 255] bw ( KiB/s): min= 1312, max=16328, per=99.96%, avg=3058.78, stdev=2130.22, samples=240 iops : min= 656, max= 8164, avg=1529.39, stdev=1065.11, samples=240 lat (usec) : 2=0.01%, 750=0.03%, 1000=0.03% lat (msec) : 2=0.01%, 4=0.15%, 10=1.10%, 20=2.09%, 50=61.12% lat (msec) : 100=2.95%, 250=32.48%, 500=0.03% cpu : usr=0.22%, sys=1.54%, ctx=136623, majf=0, minf=1 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,183635,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=3061KiB/s (3134kB/s), 3061KiB/s-3061KiB/s (3134kB/s-3134kB/s), io=359MiB (376MB), run=120001-120001msec

Disk stats (read/write): nvme1n1: ios=31/285820, merge=0/372, ticks=28/1388315, in_queue=1388343, util=99.86%

[wanghenshui]

sudo fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=4096 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=5916KiB/s][r=0,w=1479 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=15261: Tue Apr 23 15:11:03 2024 write: IOPS=1883, BW=7532KiB/s (7713kB/s)(883MiB/120002msec) slat (nsec): min=1097, max=96025k, avg=530436.61, stdev=3304056.09 clat (usec): min=626, max=150995, avg=52574.89, stdev=24532.63 lat (usec): min=1766, max=150997, avg=53105.40, stdev=24640.55 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 59], 80.00th=[ 83], 90.00th=[ 90], 95.00th=[ 96], | 99.00th=[ 107], 99.50th=[ 113], 99.90th=[ 126], 99.95th=[ 142], | 99.99th=[ 150] bw ( KiB/s): min= 4792, max=32784, per=99.96%, avg=7528.72, stdev=2870.63, samples=240 iops : min= 1198, max= 8196, avg=1882.18, stdev=717.66, samples=240 lat (usec) : 750=0.02%, 1000=0.03% lat (msec) : 2=0.01%, 4=0.16%, 10=0.93%, 20=0.91%, 50=62.35% lat (msec) : 100=33.00%, 250=2.59% cpu : usr=0.12%, sys=0.61%, ctx=18608, majf=0, minf=1 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,225965,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=7532KiB/s (7713kB/s), 7532KiB/s-7532KiB/s (7713kB/s-7713kB/s), io=883MiB (926MB), run=120002-120002msec

Disk stats (read/write): nvme1n1: ios=4/362870, merge=0/61, ticks=6/1472830, in_queue=1472836, util=99.95%

[wanghenshui]

sudo fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=8192 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=11.9MiB/s][r=0,w=1517 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=16567: Tue Apr 23 15:13:56 2024 write: IOPS=1962, BW=15.3MiB/s (16.1MB/s)(1840MiB/120037msec) slat (nsec): min=1240, max=62931k, avg=508918.07, stdev=3227368.98 clat (usec): min=652, max=145296, avg=50457.49, stdev=24035.18 lat (usec): min=1287, max=145298, avg=50966.47, stdev=24140.15 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 40], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 47], | 70.00th=[ 53], 80.00th=[ 80], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 123], 99.95th=[ 129], | 99.99th=[ 146] bw ( KiB/s): min= 9520, max=66480, per=99.98%, avg=15693.57, stdev=5959.86, samples=240 iops : min= 1190, max= 8310, avg=1961.68, stdev=744.98, samples=240 lat (usec) : 750=0.01%, 1000=0.03% lat (msec) : 2=0.02%, 4=0.43%, 10=1.01%, 20=0.81%, 50=65.29% lat (msec) : 100=30.16%, 250=2.24% cpu : usr=0.15%, sys=0.70%, ctx=20133, majf=0, minf=1 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,235517,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=15.3MiB/s (16.1MB/s), 15.3MiB/s-15.3MiB/s (16.1MB/s-16.1MB/s), io=1840MiB (1929MB), run=120037-120037msec

Disk stats (read/write): nvme1n1: ios=0/362682, merge=0/117, ticks=0/1599358, in_queue=1599358, util=99.95%

[wanghenshui]

sudo fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=16384 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=23.0MiB/s][r=0,w=1473 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=20952: Tue Apr 23 15:20:09 2024 write: IOPS=1969, BW=30.8MiB/s (32.3MB/s)(3694MiB/120035msec) slat (nsec): min=1428, max=94430k, avg=507098.17, stdev=3409849.31 clat (usec): min=790, max=195078, avg=50267.20, stdev=23815.53 lat (usec): min=1175, max=195081, avg=50774.36, stdev=23939.59 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 24], 10.00th=[ 25], 20.00th=[ 28], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 53], 80.00th=[ 78], 90.00th=[ 89], 95.00th=[ 94], | 99.00th=[ 107], 99.50th=[ 112], 99.90th=[ 136], 99.95th=[ 148], | 99.99th=[ 194] bw ( KiB/s): min=19264, max=135072, per=99.98%, avg=31504.77, stdev=11811.77, samples=240 iops : min= 1204, max= 8442, avg=1969.03, stdev=738.24, samples=240 lat (usec) : 1000=0.01% lat (msec) : 2=0.07%, 4=0.77%, 10=0.54%, 20=0.82%, 50=62.19% lat (msec) : 100=33.24%, 250=2.37% cpu : usr=0.23%, sys=0.74%, ctx=22418, majf=0, minf=1 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,236405,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=30.8MiB/s (32.3MB/s), 30.8MiB/s-30.8MiB/s (32.3MB/s-32.3MB/s), io=3694MiB (3873MB), run=120035-120035msec

Disk stats (read/write): nvme1n1: ios=0/363569, merge=0/1922, ticks=0/1568433, in_queue=1568434, util=99.93%

[wanghenshui]

bad

fio -filename=./10G.data -iodepth 100 -thread -rw=write -ioengine=libaio --direct=1 -sync=1 -bs=8192 -size=10G -numjobs=1 -runtime=120 -group_reporting -name=mytest mytest: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=100 fio-3.7 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=44.7MiB/s][r=0,w=5723 IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=107192: Tue Apr 23 07:16:09 2024 write: IOPS=5738, BW=44.8MiB/s (47.0MB/s)(5380MiB/120006msec) slat (usec): min=2, max=17966, avg=61.26, stdev=452.45 clat (usec): min=889, max=193724, avg=17363.72, stdev=6756.18 lat (usec): min=904, max=194442, avg=17425.36, stdev=6730.68 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 38], 99.50th=[ 44], 99.90th=[ 62], 99.95th=[ 85], | 99.99th=[ 120] bw ( KiB/s): min=39312, max=86288, per=99.98%, avg=45897.67, stdev=3548.74, samples=240 iops : min= 4914, max=10786, avg=5737.20, stdev=443.60, samples=240 lat (usec) : 1000=0.01% lat (msec) : 2=0.22%, 4=1.39%, 10=8.76%, 20=66.86%, 50=22.51% lat (msec) : 100=0.22%, 250=0.04% cpu : usr=1.08%, sys=3.67%, ctx=57450, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,688623,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs): WRITE: bw=44.8MiB/s (47.0MB/s), 44.8MiB/s-44.8MiB/s (47.0MB/s-47.0MB/s), io=5380MiB (5641MB), run=120006-120006msec

Disk stats (read/write): vdb: ios=0/724840, merge=0/6, ticks=0/4518238, in_queue=4518332, util=98.88%