search

hejing / instance_containize

any issues when using the bitdeer.ai
0 stars 0 forks source link

mount vs native disk compare in the IO performance #9

Open jianwang-ntu opened 1 month ago

jianwang-ntu commented 1 month ago 
(base) ➜  /home  df -h /home
Filesystem      Size  Used Avail Use% Mounted on
overlay          97G   71G   27G  73% /

(base) ➜   /home  cd /home
(base) ➜   /home  bash testio.sh

Testing: Random read/write performance
randreadwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
Jobs: 1 (f=1): [_(2),m(1),_(5)][100.0%][r=10.7MiB/s,w=3784KiB/s][r=2737,w=946 IOPS][eta 00m:00s]          
randreadwrite: (groupid=0, jobs=8): err= 0: pid=139878: Sat Sep 28 08:56:18 2024
  read: IOPS=10.6k, BW=41.3MiB/s (43.3MB/s)(3071MiB/74288msec)
    slat (usec): min=2, max=3322, avg= 9.76, stdev=14.34
    clat (nsec): min=642, max=1114.5M, avg=403793.46, stdev=1948686.71
     lat (usec): min=188, max=1114.5k, avg=413.72, stdev=1948.76
    clat percentiles (usec):
     |  1.00th=[  229],  5.00th=[  243], 10.00th=[  253], 20.00th=[  265],
     | 30.00th=[  281], 40.00th=[  293], 50.00th=[  318], 60.00th=[  338],
     | 70.00th=[  359], 80.00th=[  392], 90.00th=[  502], 95.00th=[  644],
     | 99.00th=[ 1565], 99.50th=[ 4228], 99.90th=[ 9503], 99.95th=[10945],
     | 99.99th=[16909]
   bw (  KiB/s): min=12504, max=61048, per=100.00%, avg=43230.78, stdev=1945.16, samples=1158
   iops        : min= 3126, max=15262, avg=10807.68, stdev=486.29, samples=1158
  write: IOPS=3533, BW=13.8MiB/s (14.5MB/s)(1025MiB/74288msec); 0 zone resets
    slat (usec): min=2, max=1858, avg=10.94, stdev=12.28
    clat (usec): min=260, max=87475, avg=965.06, stdev=2976.63
     lat (usec): min=299, max=87488, avg=976.17, stdev=2976.90
    clat percentiles (usec):
     |  1.00th=[  338],  5.00th=[  359], 10.00th=[  371], 20.00th=[  392],
     | 30.00th=[  416], 40.00th=[  461], 50.00th=[  502], 60.00th=[  562],
     | 70.00th=[  709], 80.00th=[  922], 90.00th=[ 1139], 95.00th=[ 1336],
     | 99.00th=[17171], 99.50th=[26346], 99.90th=[39584], 99.95th=[43254],
     | 99.99th=[50594]
   bw (  KiB/s): min= 4176, max=21063, per=100.00%, avg=14429.54, stdev=649.58, samples=1158
   iops        : min= 1044, max= 5265, avg=3607.37, stdev=162.39, samples=1158
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=6.62%, 500=73.21%, 750=10.57%, 1000=4.02%
  lat (msec)   : 2=4.51%, 4=0.29%, 10=0.38%, 20=0.19%, 50=0.21%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.74%, sys=2.84%, ctx=1048778, majf=0, minf=166
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=786102,262474,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.3MiB/s (43.3MB/s), 41.3MiB/s-41.3MiB/s (43.3MB/s-43.3MB/s), io=3071MiB (3220MB), run=74288-74288msec
  WRITE: bw=13.8MiB/s (14.5MB/s), 13.8MiB/s-13.8MiB/s (14.5MB/s-14.5MB/s), io=1025MiB (1075MB), run=74288-74288msec

Testing: Random read performance
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
Jobs: 1 (f=1): [_(1),r(1),_(6)][100.0%][r=8192KiB/s][r=2048 IOPS][eta 00m:00s]         
randread: (groupid=0, jobs=8): err= 0: pid=139888: Sat Sep 28 08:57:11 2024
  read: IOPS=22.0k, BW=85.9MiB/s (90.1MB/s)(4096MiB/47677msec)
    slat (usec): min=2, max=323, avg= 8.20, stdev= 3.57
    clat (nsec): min=1415, max=27108k, avg=318328.59, stdev=166667.08
     lat (usec): min=181, max=27112, avg=326.67, stdev=166.74
    clat percentiles (usec):
     |  1.00th=[  217],  5.00th=[  229], 10.00th=[  237], 20.00th=[  247],
     | 30.00th=[  255], 40.00th=[  265], 50.00th=[  277], 60.00th=[  289],
     | 70.00th=[  314], 80.00th=[  347], 90.00th=[  429], 95.00th=[  562],
     | 99.00th=[  906], 99.50th=[ 1139], 99.90th=[ 2089], 99.95th=[ 2442],
     | 99.99th=[ 3621]
   bw (  KiB/s): min=85968, max=107760, per=100.00%, avg=97967.49, stdev=586.81, samples=682
   iops        : min=21492, max=26940, avg=24491.81, stdev=146.72, samples=682
  lat (usec)   : 2=0.01%, 100=0.01%, 250=24.12%, 500=69.08%, 750=5.10%
  lat (usec)   : 1000=1.01%
  lat (msec)   : 2=0.57%, 4=0.11%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.08%, sys=4.08%, ctx=1048664, majf=0, minf=102
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=85.9MiB/s (90.1MB/s), 85.9MiB/s-85.9MiB/s (90.1MB/s-90.1MB/s), io=4096MiB (4295MB), run=47677-47677msec

Testing: Random write performance
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=62.8MiB/s][w=16.1k IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=8): err= 0: pid=139971: Sat Sep 28 08:58:19 2024
  write: IOPS=15.5k, BW=60.5MiB/s (63.4MB/s)(4096MiB/67746msec); 0 zone resets
    slat (usec): min=6, max=38124, avg=18.89, stdev=135.66
    clat (nsec): min=1805, max=41457k, avg=493861.83, stdev=354163.72
     lat (usec): min=304, max=41477, avg=512.92, stdev=380.55
    clat percentiles (usec):
     |  1.00th=[  338],  5.00th=[  351], 10.00th=[  363], 20.00th=[  375],
     | 30.00th=[  388], 40.00th=[  404], 50.00th=[  433], 60.00th=[  474],
     | 70.00th=[  510], 80.00th=[  562], 90.00th=[  652], 95.00th=[  758],
     | 99.00th=[ 1401], 99.50th=[ 1762], 99.90th=[ 2671], 99.95th=[ 3556],
     | 99.99th=[14222]
   bw (  KiB/s): min=26088, max=68040, per=100.00%, avg=62296.30, stdev=669.89, samples=1073
   iops        : min= 6522, max=17010, avg=15574.07, stdev=167.47, samples=1073
  lat (usec)   : 2=0.01%, 250=0.01%, 500=67.22%, 750=27.59%, 1000=3.15%
  lat (msec)   : 2=1.73%, 4=0.27%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=0.73%, sys=4.64%, ctx=1048968, majf=0, minf=95
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=60.5MiB/s (63.4MB/s), 60.5MiB/s-60.5MiB/s (63.4MB/s-63.4MB/s), io=4096MiB (4295MB), run=67746-67746msec

Testing: IO Latency on individual request
4 KiB <<< . (overlay overlay 96.7 GiB): request=1 time=3.74 ms (warmup)
4 KiB <<< . (overlay overlay 96.7 GiB): request=2 time=1.34 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=3 time=1.04 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=4 time=1.25 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=5 time=1.39 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=6 time=1.22 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=7 time=1.46 ms (slow)
4 KiB <<< . (overlay overlay 96.7 GiB): request=8 time=1.28 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=9 time=1.93 ms (slow)
4 KiB <<< . (overlay overlay 96.7 GiB): request=10 time=1.54 ms

--- . (overlay overlay 96.7 GiB) ioping statistics ---
9 requests completed in 12.5 ms, 36 KiB read, 722 iops, 2.82 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 1.04 ms / 1.38 ms / 1.93 ms / 236.5 us
jianwang-ntu commented 1 month ago

When we test performance on the network disk, the IO drop significantly . 

(base) ➜  ~ df -h /home/jovyan 
Filesystem                                                                                                                                      Size  Used Avail Use% Mounted on
10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c  800G  745G   56G  94% /home/jovyan

(base) ➜  ~ cd /home/jovyan 
(base) ➜  ~ bash testio.sh 
Testing: Random read/write performance
randreadwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=5373KiB/s,w=1773KiB/s][r=1343,w=443 IOPS][eta 00m:00s]
randreadwrite: (groupid=0, jobs=8): err= 0: pid=140036: Sat Sep 28 09:12:14 2024
  read: IOPS=1429, BW=5717KiB/s (5854kB/s)(1340MiB/240009msec)
    slat (usec): min=8, max=2754, avg=43.54, stdev=24.12
    clat (usec): min=514, max=157748, avg=4146.01, stdev=4565.81
     lat (usec): min=546, max=157807, avg=4189.90, stdev=4566.06
    clat percentiles (usec):
     |  1.00th=[  799],  5.00th=[  955], 10.00th=[ 1057], 20.00th=[ 1188],
     | 30.00th=[ 1270], 40.00th=[ 1369], 50.00th=[ 1516], 60.00th=[ 3097],
     | 70.00th=[ 5407], 80.00th=[ 7701], 90.00th=[ 9896], 95.00th=[11863],
     | 99.00th=[18744], 99.50th=[23200], 99.90th=[37487], 99.95th=[47449],
     | 99.99th=[78119]
   bw (  KiB/s): min= 2216, max= 8224, per=100.00%, avg=5719.83, stdev=111.66, samples=3832
   iops        : min=  554, max= 2056, avg=1429.95, stdev=27.91, samples=3832
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(448MiB/240009msec); 0 zone resets
    slat (usec): min=9, max=435, avg=49.95, stdev=27.30
    clat (usec): min=1255, max=237993, avg=4149.47, stdev=8256.68
     lat (usec): min=1325, max=238155, avg=4199.78, stdev=8257.19
    clat percentiles (usec):
     |  1.00th=[  1827],  5.00th=[  2040], 10.00th=[  2147], 20.00th=[  2278],
     | 30.00th=[  2376], 40.00th=[  2442], 50.00th=[  2540], 60.00th=[  2638],
     | 70.00th=[  2737], 80.00th=[  3064], 90.00th=[  6521], 95.00th=[ 10683],
     | 99.00th=[ 30278], 99.50th=[ 67634], 99.90th=[119014], 99.95th=[128451],
     | 99.99th=[160433]
   bw (  KiB/s): min=  704, max= 3280, per=100.00%, avg=1911.87, stdev=52.10, samples=3832
   iops        : min=  176, max=  820, avg=477.95, stdev=13.03, samples=3832
  lat (usec)   : 750=0.44%, 1000=4.99%
  lat (msec)   : 2=37.79%, 4=25.88%, 10=22.32%, 20=7.57%, 50=0.81%
  lat (msec)   : 100=0.14%, 250=0.06%
  cpu          : usr=0.22%, sys=1.07%, ctx=666983, majf=0, minf=127
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=343008,114663,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5717KiB/s (5854kB/s), 5717KiB/s-5717KiB/s (5854kB/s-5854kB/s), io=1340MiB (1405MB), run=240009-240009msec
  WRITE: bw=1911KiB/s (1957kB/s), 1911KiB/s-1911KiB/s (1957kB/s-1957kB/s), io=448MiB (470MB), run=240009-240009msec

Testing: Random read performance
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=10.3MiB/s][r=2645 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=8): err= 0: pid=140181: Sat Sep 28 09:16:48 2024
  read: IOPS=2628, BW=10.3MiB/s (10.8MB/s)(2464MiB/240005msec)
    slat (usec): min=7, max=4125, avg=42.86, stdev=22.26
    clat (usec): min=432, max=143128, avg=2998.37, stdev=3461.03
     lat (usec): min=488, max=143158, avg=3041.56, stdev=3461.13
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[  865], 10.00th=[  947], 20.00th=[ 1074],
     | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1401],
     | 70.00th=[ 2343], 80.00th=[ 5276], 90.00th=[ 8291], 95.00th=[ 9896],
     | 99.00th=[14746], 99.50th=[17695], 99.90th=[27395], 99.95th=[32375],
     | 99.99th=[43254]
   bw (  KiB/s): min= 5360, max=14104, per=100.00%, avg=10519.42, stdev=145.11, samples=3832
   iops        : min= 1340, max= 3526, avg=2629.85, stdev=36.28, samples=3832
  lat (usec)   : 500=0.01%, 750=1.42%, 1000=12.40%
  lat (msec)   : 2=55.47%, 4=6.27%, 10=19.89%, 20=4.23%, 50=0.31%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.25%, sys=1.35%, ctx=978340, majf=0, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=630793,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=10.3MiB/s (10.8MB/s), 10.3MiB/s-10.3MiB/s (10.8MB/s-10.8MB/s), io=2464MiB (2584MB), run=240005-240005msec

Testing: Random write performance
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=4256KiB/s][w=1064 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=8): err= 0: pid=140475: Sat Sep 28 09:20:48 2024
  write: IOPS=1329, BW=5320KiB/s (5447kB/s)(1247MiB/240076msec); 0 zone resets
    slat (usec): min=9, max=23259, avg=49.30, stdev=113.73
    clat (usec): min=987, max=270938, avg=5961.55, stdev=15564.56
     lat (usec): min=1003, max=270969, avg=6011.19, stdev=15564.97
    clat percentiles (usec):
     |  1.00th=[  1565],  5.00th=[  1778], 10.00th=[  1893], 20.00th=[  2057],
     | 30.00th=[  2180], 40.00th=[  2278], 50.00th=[  2376], 60.00th=[  2507],
     | 70.00th=[  2638], 80.00th=[  2933], 90.00th=[  6652], 95.00th=[ 19268],
     | 99.00th=[ 98042], 99.50th=[116917], 99.90th=[137364], 99.95th=[145753],
     | 99.99th=[214959]
   bw (  KiB/s): min= 1392, max= 9754, per=100.00%, avg=5325.46, stdev=176.57, samples=3832
   iops        : min=  348, max= 2438, avg=1331.23, stdev=44.13, samples=3832
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=16.19%, 4=70.71%, 10=4.70%, 20=3.53%, 50=2.07%
  lat (msec)   : 100=1.85%, 250=0.94%, 500=0.01%
  cpu          : usr=0.15%, sys=0.78%, ctx=465152, majf=0, minf=79
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,319282,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5320KiB/s (5447kB/s), 5320KiB/s-5320KiB/s (5447kB/s-5447kB/s), io=1247MiB (1308MB), run=240076-240076msec

Testing: IO Latency on individual request
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=1 time=1.54 ms (warmup)
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=2 time=1.98 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=3 time=1.67 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=4 time=1.71 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=5 time=1.64 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=6 time=1.63 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=7 time=1.63 ms (fast)
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=8 time=1.66 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=9 time=1.54 ms (fast)
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=10 time=1.47 ms (fast)

--- . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB) ioping statistics ---
9 requests completed in 14.9 ms, 36 KiB read, 602 iops, 2.36 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 1.47 ms / 1.66 ms / 1.98 ms / 132.2 us
jianwang-ntu commented 1 month ago

The tool can be found from https://gist.github.com/jsandas/a3705a619214470b2d92b1fb2b559ad9

jianwang-ntu commented 1 month ago

8X h100 in /home/ubuntu/.cache/huggingface/hub

Testing: Random read/write performance randreadwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.28 Starting 8 processes randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) Jobs: 1 (f=1): [_(7),m(1)][100.0%][r=13.2MiB/s,w=4664KiB/s][r=3371,w=1166 IOPS][eta 00m:00s] randreadwrite: (groupid=0, jobs=8): err= 0: pid=82784: Thu Oct 3 07:12:04 2024 read: IOPS=29.0k, BW=113MiB/s (119MB/s)(3071MiB/27090msec) slat (usec): min=2, max=263, avg= 9.41, stdev= 5.38 clat (nsec): min=537, max=2818.6k, avg=185967.44, stdev=49727.41 lat (usec): min=89, max=2823, avg=195.46, stdev=50.93 clat percentiles (usec): | 1.00th=[ 115], 5.00th=[ 125], 10.00th=[ 131], 20.00th=[ 139], | 30.00th=[ 147], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 196], | 70.00th=[ 204], 80.00th=[ 235], 90.00th=[ 251], 95.00th=[ 260], | 99.00th=[ 297], 99.50th=[ 322], 99.90th=[ 441], 99.95th=[ 490], | 99.99th=[ 1401] bw ( KiB/s): min=120368, max=135530, per=100.00%, avg=127625.36, stdev=508.92, samples=391 iops : min=30092, max=33882, avg=31906.33, stdev=127.23, samples=391 write: IOPS=9688, BW=37.8MiB/s (39.7MB/s)(1025MiB/27090msec); 0 zone resets slat (usec): min=2, max=298, avg= 9.73, stdev= 5.56 clat (usec): min=48, max=10520, avg=155.34, stdev=90.83 lat (usec): min=56, max=10537, avg=165.16, stdev=91.58 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 115], 10.00th=[ 121], 20.00th=[ 128], | 30.00th=[ 135], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 198], 95.00th=[ 208], | 99.00th=[ 255], 99.50th=[ 262], 99.90th=[ 396], 99.95th=[ 1123], | 99.99th=[ 4555] bw ( KiB/s): min=38496, max=47152, per=100.00%, avg=42610.01, stdev=244.53, samples=391 iops : min= 9624, max=11788, avg=10652.49, stdev=61.13, samples=391 lat (nsec) : 750=0.01% lat (usec) : 50=0.01%, 100=0.65%, 250=91.38%, 500=7.92%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=0.94%, sys=6.73%, ctx=1048699, majf=0, minf=138 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=786102,262474,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=113MiB/s (119MB/s), 113MiB/s-113MiB/s (119MB/s-119MB/s), io=3071MiB (3220MB), run=27090-27090msec WRITE: bw=37.8MiB/s (39.7MB/s), 37.8MiB/s-37.8MiB/s (39.7MB/s-39.7MB/s), io=1025MiB (1075MB), run=27090-27090msec

Testing: Random read performance randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.28 Starting 8 processes randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) Jobs: 1 (f=1): [_(7),r(1)][100.0%][r=17.4MiB/s][r=4464 IOPS][eta 00m:00s] randread: (groupid=0, jobs=8): err= 0: pid=82794: Thu Oct 3 07:12:44 2024 read: IOPS=30.4k, BW=119MiB/s (124MB/s)(4096MiB/34535msec) slat (nsec): min=1929, max=187545, avg=10570.20, stdev=5972.98 clat (usec): min=72, max=6597, avg=235.42, stdev=43.85 lat (usec): min=75, max=6601, avg=246.08, stdev=45.04 clat percentiles (usec): | 1.00th=[ 122], 5.00th=[ 163], 10.00th=[ 182], 20.00th=[ 202], | 30.00th=[ 233], 40.00th=[ 241], 50.00th=[ 245], 60.00th=[ 251], | 70.00th=[ 255], 80.00th=[ 262], 90.00th=[ 269], 95.00th=[ 277], | 99.00th=[ 306], 99.50th=[ 326], 99.90th=[ 420], 99.95th=[ 594], | 99.99th=[ 1582] bw ( KiB/s): min=124744, max=136056, per=100.00%, avg=129899.90, stdev=321.41, samples=513 iops : min=31186, max=34014, avg=32474.94, stdev=80.36, samples=513 lat (usec) : 100=0.01%, 250=59.73%, 500=40.20%, 750=0.04%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01% cpu : usr=0.69%, sys=5.95%, ctx=1048690, majf=0, minf=103 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=119MiB/s (124MB/s), 119MiB/s-119MiB/s (124MB/s-124MB/s), io=4096MiB (4295MB), run=34535-34535msec

Testing: Random write performance randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.28 Starting 8 processes randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) Jobs: 7 (f=7): [_(1),w(7)][98.0%][w=87.0MiB/s][w=22.3k IOPS][eta 00m:01s] randwrite: (groupid=0, jobs=8): err= 0: pid=82804: Thu Oct 3 07:13:33 2024 write: IOPS=21.2k, BW=82.8MiB/s (86.8MB/s)(4096MiB/49494msec); 0 zone resets slat (usec): min=4, max=17606, avg=21.15, stdev=49.11 clat (usec): min=66, max=1829.1k, avg=352.70, stdev=5058.04 lat (usec): min=71, max=1829.1k, avg=373.96, stdev=5058.28 clat percentiles (usec): | 1.00th=[ 180], 5.00th=[ 239], 10.00th=[ 247], 20.00th=[ 255], | 30.00th=[ 265], 40.00th=[ 289], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 363], 80.00th=[ 375], 90.00th=[ 408], 95.00th=[ 429], | 99.00th=[ 914], 99.50th=[ 1352], 99.90th=[ 3425], 99.95th=[ 5211], | 99.99th=[13435] bw ( KiB/s): min=44784, max=116736, per=100.00%, avg=87920.95, stdev=1946.58, samples=758 iops : min=11196, max=29184, avg=21980.24, stdev=486.65, samples=758 lat (usec) : 100=0.01%, 250=14.14%, 500=83.59%, 750=0.98%, 1000=0.40% lat (msec) : 2=0.64%, 4=0.16%, 10=0.06%, 20=0.02%, 50=0.01% lat (msec) : 2000=0.01% cpu : usr=0.43%, sys=6.06%, ctx=1048685, majf=0, minf=118 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=82.8MiB/s (86.8MB/s), 82.8MiB/s-82.8MiB/s (86.8MB/s-86.8MB/s), io=4096MiB (4295MB), run=49494-49494msec

Testing: IO Latency on individual request 4 KiB <<< . (overlay overlay 1.94 TiB): request=1 time=271.0 us (warmup) 4 KiB <<< . (overlay overlay 1.94 TiB): request=2 time=328.7 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=3 time=334.7 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=4 time=370.6 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=5 time=351.8 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=6 time=336.1 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=7 time=273.4 us (fast) 4 KiB <<< . (overlay overlay 1.94 TiB): request=8 time=297.1 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=9 time=283.0 us 4 KiB <<< . (overlay overlay 1.94 TiB): request=10 time=320.0 us

--- . (overlay overlay 1.94 TiB) ioping statistics --- 9 requests completed in 2.90 ms, 36 KiB read, 3.11 k iops, 12.1 MiB/s generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s min/avg/max/mdev = 273.4 us / 321.7 us / 370.6 us / 30.1 us

jianwang-ntu commented 1 month ago

8XA100 on .cache/huggingface/hub

Testing: Random read/write performance randreadwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.16 Starting 8 processes randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) randreadwrite: Laying out IO file (1 file / 512MiB) Jobs: 1 (f=1): [(1),m(1),(6)][98.5%][r=23.5MiB/s,w=7936KiB/s][r=6008,w=1984 IOPS][eta 00m:01s] randreadwrite: (groupid=0, jobs=8): err= 0: pid=1709440: Thu Oct 3 07:13:04 2024 read: IOPS=12.4k, BW=48.3MiB/s (50.6MB/s)(3071MiB/63602msec) slat (usec): min=2, max=2902, avg= 8.45, stdev=13.46 clat (usec): min=9, max=28855, avg=471.41, stdev=253.22 lat (usec): min=188, max=28863, avg=479.96, stdev=254.82 clat percentiles (usec): | 1.00th=[ 223], 5.00th=[ 249], 10.00th=[ 265], 20.00th=[ 297], | 30.00th=[ 326], 40.00th=[ 363], 50.00th=[ 424], 60.00th=[ 506], | 70.00th=[ 562], 80.00th=[ 611], 90.00th=[ 693], 95.00th=[ 775], | 99.00th=[ 1156], 99.50th=[ 1516], 99.90th=[ 3326], 99.95th=[ 4228], | 99.99th=[ 5669] bw ( KiB/s): min=43304, max=71513, per=100.00%, avg=51324.84, stdev=594.67, samples=975 iops : min=10826, max=17877, avg=12830.90, stdev=148.63, samples=975 write: IOPS=4126, BW=16.1MiB/s (16.9MB/s)(1025MiB/63602msec); 0 zone resets slat (usec): min=2, max=4205, avg= 9.28, stdev=19.01 clat (usec): min=210, max=24257, avg=411.65, stdev=178.17 lat (usec): min=226, max=24265, avg=421.03, stdev=181.37 clat percentiles (usec): | 1.00th=[ 260], 5.00th=[ 281], 10.00th=[ 297], 20.00th=[ 322], | 30.00th=[ 343], 40.00th=[ 363], 50.00th=[ 383], 60.00th=[ 408], | 70.00th=[ 437], 80.00th=[ 474], 90.00th=[ 537], 95.00th=[ 603], | 99.00th=[ 816], 99.50th=[ 1057], 99.90th=[ 2802], 99.95th=[ 3785], | 99.99th=[ 5342] bw ( KiB/s): min=13680, max=24193, per=100.00%, avg=17132.60, stdev=219.89, samples=975 iops : min= 3420, max= 6046, avg=4282.81, stdev=54.94, samples=975 lat (usec) : 10=0.01%, 50=0.01%, 100=0.01%, 250=4.21%, 500=61.21% lat (usec) : 750=29.58%, 1000=3.74% lat (msec) : 2=1.01%, 4=0.19%, 10=0.06%, 20=0.01%, 50=0.01% cpu : usr=0.52%, sys=3.12%, ctx=1049904, majf=0, minf=181 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=786102,262474,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=48.3MiB/s (50.6MB/s), 48.3MiB/s-48.3MiB/s (50.6MB/s-50.6MB/s), io=3071MiB (3220MB), run=63602-63602msec WRITE: bw=16.1MiB/s (16.9MB/s), 16.1MiB/s-16.1MiB/s (16.9MB/s-16.9MB/s), io=1025MiB (1075MB), run=63602-63602msec

Testing: Random read performance randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.16 Starting 8 processes randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) randread: Laying out IO file (1 file / 512MiB) Jobs: 4 (f=3): [r(2),f(1),r(1),_(4)][100.0%][r=49.6MiB/s][r=12.7k IOPS][eta 00m:00s] randread: (groupid=0, jobs=8): err= 0: pid=1709706: Thu Oct 3 07:14:19 2024 read: IOPS=16.1k, BW=62.9MiB/s (66.0MB/s)(4096MiB/65068msec) slat (usec): min=2, max=2566, avg= 9.25, stdev=10.16 clat (usec): min=12, max=31182, avg=472.26, stdev=240.65 lat (usec): min=187, max=31192, avg=481.61, stdev=241.82 clat percentiles (usec): | 1.00th=[ 227], 5.00th=[ 251], 10.00th=[ 269], 20.00th=[ 302], | 30.00th=[ 334], 40.00th=[ 375], 50.00th=[ 453], 60.00th=[ 515], | 70.00th=[ 553], 80.00th=[ 603], 90.00th=[ 676], 95.00th=[ 750], | 99.00th=[ 1074], 99.50th=[ 1418], 99.90th=[ 3195], 99.95th=[ 4228], | 99.99th=[ 5538] bw ( KiB/s): min=50946, max=80766, per=100.00%, avg=65346.01, stdev=792.80, samples=1005 iops : min=12735, max=20191, avg=16336.19, stdev=198.20, samples=1005 lat (usec) : 20=0.01%, 100=0.01%, 250=4.75%, 500=52.49%, 750=37.71% lat (usec) : 1000=3.86% lat (msec) : 2=0.95%, 4=0.18%, 10=0.06%, 20=0.01%, 50=0.01% cpu : usr=0.47%, sys=3.12%, ctx=1049767, majf=0, minf=165 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=62.9MiB/s (66.0MB/s), 62.9MiB/s-62.9MiB/s (66.0MB/s-66.0MB/s), io=4096MiB (4295MB), run=65068-65068msec

Testing: Random write performance randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.16 Starting 8 processes randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) Jobs: 4 (f=3): [(3),f(1),w(2),(1),w(1)][100.0%][w=67.4MiB/s][w=17.2k IOPS][eta 00m:00s] randwrite: (groupid=0, jobs=8): err= 0: pid=1709972: Thu Oct 3 07:15:20 2024 write: IOPS=17.5k, BW=68.4MiB/s (71.7MB/s)(4096MiB/59913msec); 0 zone resets slat (usec): min=2, max=3783, avg=10.46, stdev=12.57 clat (usec): min=8, max=35004, avg=443.67, stdev=204.94 lat (usec): min=222, max=35013, avg=454.23, stdev=206.64 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 293], 10.00th=[ 314], 20.00th=[ 343], | 30.00th=[ 363], 40.00th=[ 383], 50.00th=[ 408], 60.00th=[ 433], | 70.00th=[ 469], 80.00th=[ 519], 90.00th=[ 594], 95.00th=[ 668], | 99.00th=[ 988], 99.50th=[ 1254], 99.90th=[ 2868], 99.95th=[ 4178], | 99.99th=[ 5800] bw ( KiB/s): min=61416, max=79296, per=100.00%, avg=70228.38, stdev=403.24, samples=948 iops : min=15354, max=19824, avg=17556.90, stdev=100.81, samples=948 lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.22%, 500=76.76% lat (usec) : 750=20.38%, 1000=1.67% lat (msec) : 2=0.79%, 4=0.12%, 10=0.06%, 20=0.01%, 50=0.01% cpu : usr=0.74%, sys=3.35%, ctx=1049691, majf=0, minf=217 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=68.4MiB/s (71.7MB/s), 68.4MiB/s-68.4MiB/s (71.7MB/s-71.7MB/s), io=4096MiB (4295MB), run=59913-59913msec

Testing: IO Latency on individual request 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=1 time=381.7 us (warmup) 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=2 time=705.8 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=3 time=580.1 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=4 time=709.7 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=5 time=697.4 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=6 time=651.5 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=7 time=670.8 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=8 time=591.4 us 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=9 time=763.6 us (slow) 4 KiB <<< . (nfs 192.168.128.103:/server8/wangjian): request=10 time=548.2 us (fast)

--- . (nfs 192.168.128.103:/server8/wangjian) ioping statistics --- 9 requests completed in 5.92 ms, 36 KiB read, 1.52 k iops, 5.94 MiB/s generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s min/avg/max/mdev = 548.2 us / 657.6 us / 763.6 us / 67 us