search

hejing / instance_containize

any issues when using the bitdeer.ai
0 stars 0 forks source link

mount vs native disk compare in the IO performance #9

Open jianwang-ntu opened 2 days ago

jianwang-ntu commented 2 days ago 
(base) ➜  /home  df -h /home
Filesystem      Size  Used Avail Use% Mounted on
overlay          97G   71G   27G  73% /

(base) ➜   /home  cd /home
(base) ➜   /home  bash testio.sh

Testing: Random read/write performance
randreadwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
Jobs: 1 (f=1): [_(2),m(1),_(5)][100.0%][r=10.7MiB/s,w=3784KiB/s][r=2737,w=946 IOPS][eta 00m:00s]          
randreadwrite: (groupid=0, jobs=8): err= 0: pid=139878: Sat Sep 28 08:56:18 2024
  read: IOPS=10.6k, BW=41.3MiB/s (43.3MB/s)(3071MiB/74288msec)
    slat (usec): min=2, max=3322, avg= 9.76, stdev=14.34
    clat (nsec): min=642, max=1114.5M, avg=403793.46, stdev=1948686.71
     lat (usec): min=188, max=1114.5k, avg=413.72, stdev=1948.76
    clat percentiles (usec):
     |  1.00th=[  229],  5.00th=[  243], 10.00th=[  253], 20.00th=[  265],
     | 30.00th=[  281], 40.00th=[  293], 50.00th=[  318], 60.00th=[  338],
     | 70.00th=[  359], 80.00th=[  392], 90.00th=[  502], 95.00th=[  644],
     | 99.00th=[ 1565], 99.50th=[ 4228], 99.90th=[ 9503], 99.95th=[10945],
     | 99.99th=[16909]
   bw (  KiB/s): min=12504, max=61048, per=100.00%, avg=43230.78, stdev=1945.16, samples=1158
   iops        : min= 3126, max=15262, avg=10807.68, stdev=486.29, samples=1158
  write: IOPS=3533, BW=13.8MiB/s (14.5MB/s)(1025MiB/74288msec); 0 zone resets
    slat (usec): min=2, max=1858, avg=10.94, stdev=12.28
    clat (usec): min=260, max=87475, avg=965.06, stdev=2976.63
     lat (usec): min=299, max=87488, avg=976.17, stdev=2976.90
    clat percentiles (usec):
     |  1.00th=[  338],  5.00th=[  359], 10.00th=[  371], 20.00th=[  392],
     | 30.00th=[  416], 40.00th=[  461], 50.00th=[  502], 60.00th=[  562],
     | 70.00th=[  709], 80.00th=[  922], 90.00th=[ 1139], 95.00th=[ 1336],
     | 99.00th=[17171], 99.50th=[26346], 99.90th=[39584], 99.95th=[43254],
     | 99.99th=[50594]
   bw (  KiB/s): min= 4176, max=21063, per=100.00%, avg=14429.54, stdev=649.58, samples=1158
   iops        : min= 1044, max= 5265, avg=3607.37, stdev=162.39, samples=1158
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=6.62%, 500=73.21%, 750=10.57%, 1000=4.02%
  lat (msec)   : 2=4.51%, 4=0.29%, 10=0.38%, 20=0.19%, 50=0.21%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.74%, sys=2.84%, ctx=1048778, majf=0, minf=166
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=786102,262474,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.3MiB/s (43.3MB/s), 41.3MiB/s-41.3MiB/s (43.3MB/s-43.3MB/s), io=3071MiB (3220MB), run=74288-74288msec
  WRITE: bw=13.8MiB/s (14.5MB/s), 13.8MiB/s-13.8MiB/s (14.5MB/s-14.5MB/s), io=1025MiB (1075MB), run=74288-74288msec

Testing: Random read performance
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
Jobs: 1 (f=1): [_(1),r(1),_(6)][100.0%][r=8192KiB/s][r=2048 IOPS][eta 00m:00s]         
randread: (groupid=0, jobs=8): err= 0: pid=139888: Sat Sep 28 08:57:11 2024
  read: IOPS=22.0k, BW=85.9MiB/s (90.1MB/s)(4096MiB/47677msec)
    slat (usec): min=2, max=323, avg= 8.20, stdev= 3.57
    clat (nsec): min=1415, max=27108k, avg=318328.59, stdev=166667.08
     lat (usec): min=181, max=27112, avg=326.67, stdev=166.74
    clat percentiles (usec):
     |  1.00th=[  217],  5.00th=[  229], 10.00th=[  237], 20.00th=[  247],
     | 30.00th=[  255], 40.00th=[  265], 50.00th=[  277], 60.00th=[  289],
     | 70.00th=[  314], 80.00th=[  347], 90.00th=[  429], 95.00th=[  562],
     | 99.00th=[  906], 99.50th=[ 1139], 99.90th=[ 2089], 99.95th=[ 2442],
     | 99.99th=[ 3621]
   bw (  KiB/s): min=85968, max=107760, per=100.00%, avg=97967.49, stdev=586.81, samples=682
   iops        : min=21492, max=26940, avg=24491.81, stdev=146.72, samples=682
  lat (usec)   : 2=0.01%, 100=0.01%, 250=24.12%, 500=69.08%, 750=5.10%
  lat (usec)   : 1000=1.01%
  lat (msec)   : 2=0.57%, 4=0.11%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.08%, sys=4.08%, ctx=1048664, majf=0, minf=102
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=85.9MiB/s (90.1MB/s), 85.9MiB/s-85.9MiB/s (90.1MB/s-90.1MB/s), io=4096MiB (4295MB), run=47677-47677msec

Testing: Random write performance
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=62.8MiB/s][w=16.1k IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=8): err= 0: pid=139971: Sat Sep 28 08:58:19 2024
  write: IOPS=15.5k, BW=60.5MiB/s (63.4MB/s)(4096MiB/67746msec); 0 zone resets
    slat (usec): min=6, max=38124, avg=18.89, stdev=135.66
    clat (nsec): min=1805, max=41457k, avg=493861.83, stdev=354163.72
     lat (usec): min=304, max=41477, avg=512.92, stdev=380.55
    clat percentiles (usec):
     |  1.00th=[  338],  5.00th=[  351], 10.00th=[  363], 20.00th=[  375],
     | 30.00th=[  388], 40.00th=[  404], 50.00th=[  433], 60.00th=[  474],
     | 70.00th=[  510], 80.00th=[  562], 90.00th=[  652], 95.00th=[  758],
     | 99.00th=[ 1401], 99.50th=[ 1762], 99.90th=[ 2671], 99.95th=[ 3556],
     | 99.99th=[14222]
   bw (  KiB/s): min=26088, max=68040, per=100.00%, avg=62296.30, stdev=669.89, samples=1073
   iops        : min= 6522, max=17010, avg=15574.07, stdev=167.47, samples=1073
  lat (usec)   : 2=0.01%, 250=0.01%, 500=67.22%, 750=27.59%, 1000=3.15%
  lat (msec)   : 2=1.73%, 4=0.27%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=0.73%, sys=4.64%, ctx=1048968, majf=0, minf=95
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=60.5MiB/s (63.4MB/s), 60.5MiB/s-60.5MiB/s (63.4MB/s-63.4MB/s), io=4096MiB (4295MB), run=67746-67746msec

Testing: IO Latency on individual request
4 KiB <<< . (overlay overlay 96.7 GiB): request=1 time=3.74 ms (warmup)
4 KiB <<< . (overlay overlay 96.7 GiB): request=2 time=1.34 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=3 time=1.04 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=4 time=1.25 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=5 time=1.39 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=6 time=1.22 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=7 time=1.46 ms (slow)
4 KiB <<< . (overlay overlay 96.7 GiB): request=8 time=1.28 ms
4 KiB <<< . (overlay overlay 96.7 GiB): request=9 time=1.93 ms (slow)
4 KiB <<< . (overlay overlay 96.7 GiB): request=10 time=1.54 ms

--- . (overlay overlay 96.7 GiB) ioping statistics ---
9 requests completed in 12.5 ms, 36 KiB read, 722 iops, 2.82 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 1.04 ms / 1.38 ms / 1.93 ms / 236.5 us
jianwang-ntu commented 2 days ago

When we test performance on the network disk, the IO drop significantly . 

(base) ➜  ~ df -h /home/jovyan 
Filesystem                                                                                                                                      Size  Used Avail Use% Mounted on
10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c  800G  745G   56G  94% /home/jovyan

(base) ➜  ~ cd /home/jovyan 
(base) ➜  ~ bash testio.sh 
Testing: Random read/write performance
randreadwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
randreadwrite: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=5373KiB/s,w=1773KiB/s][r=1343,w=443 IOPS][eta 00m:00s]
randreadwrite: (groupid=0, jobs=8): err= 0: pid=140036: Sat Sep 28 09:12:14 2024
  read: IOPS=1429, BW=5717KiB/s (5854kB/s)(1340MiB/240009msec)
    slat (usec): min=8, max=2754, avg=43.54, stdev=24.12
    clat (usec): min=514, max=157748, avg=4146.01, stdev=4565.81
     lat (usec): min=546, max=157807, avg=4189.90, stdev=4566.06
    clat percentiles (usec):
     |  1.00th=[  799],  5.00th=[  955], 10.00th=[ 1057], 20.00th=[ 1188],
     | 30.00th=[ 1270], 40.00th=[ 1369], 50.00th=[ 1516], 60.00th=[ 3097],
     | 70.00th=[ 5407], 80.00th=[ 7701], 90.00th=[ 9896], 95.00th=[11863],
     | 99.00th=[18744], 99.50th=[23200], 99.90th=[37487], 99.95th=[47449],
     | 99.99th=[78119]
   bw (  KiB/s): min= 2216, max= 8224, per=100.00%, avg=5719.83, stdev=111.66, samples=3832
   iops        : min=  554, max= 2056, avg=1429.95, stdev=27.91, samples=3832
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(448MiB/240009msec); 0 zone resets
    slat (usec): min=9, max=435, avg=49.95, stdev=27.30
    clat (usec): min=1255, max=237993, avg=4149.47, stdev=8256.68
     lat (usec): min=1325, max=238155, avg=4199.78, stdev=8257.19
    clat percentiles (usec):
     |  1.00th=[  1827],  5.00th=[  2040], 10.00th=[  2147], 20.00th=[  2278],
     | 30.00th=[  2376], 40.00th=[  2442], 50.00th=[  2540], 60.00th=[  2638],
     | 70.00th=[  2737], 80.00th=[  3064], 90.00th=[  6521], 95.00th=[ 10683],
     | 99.00th=[ 30278], 99.50th=[ 67634], 99.90th=[119014], 99.95th=[128451],
     | 99.99th=[160433]
   bw (  KiB/s): min=  704, max= 3280, per=100.00%, avg=1911.87, stdev=52.10, samples=3832
   iops        : min=  176, max=  820, avg=477.95, stdev=13.03, samples=3832
  lat (usec)   : 750=0.44%, 1000=4.99%
  lat (msec)   : 2=37.79%, 4=25.88%, 10=22.32%, 20=7.57%, 50=0.81%
  lat (msec)   : 100=0.14%, 250=0.06%
  cpu          : usr=0.22%, sys=1.07%, ctx=666983, majf=0, minf=127
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=343008,114663,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5717KiB/s (5854kB/s), 5717KiB/s-5717KiB/s (5854kB/s-5854kB/s), io=1340MiB (1405MB), run=240009-240009msec
  WRITE: bw=1911KiB/s (1957kB/s), 1911KiB/s-1911KiB/s (1957kB/s-1957kB/s), io=448MiB (470MB), run=240009-240009msec

Testing: Random read performance
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
randread: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=10.3MiB/s][r=2645 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=8): err= 0: pid=140181: Sat Sep 28 09:16:48 2024
  read: IOPS=2628, BW=10.3MiB/s (10.8MB/s)(2464MiB/240005msec)
    slat (usec): min=7, max=4125, avg=42.86, stdev=22.26
    clat (usec): min=432, max=143128, avg=2998.37, stdev=3461.03
     lat (usec): min=488, max=143158, avg=3041.56, stdev=3461.13
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[  865], 10.00th=[  947], 20.00th=[ 1074],
     | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1401],
     | 70.00th=[ 2343], 80.00th=[ 5276], 90.00th=[ 8291], 95.00th=[ 9896],
     | 99.00th=[14746], 99.50th=[17695], 99.90th=[27395], 99.95th=[32375],
     | 99.99th=[43254]
   bw (  KiB/s): min= 5360, max=14104, per=100.00%, avg=10519.42, stdev=145.11, samples=3832
   iops        : min= 1340, max= 3526, avg=2629.85, stdev=36.28, samples=3832
  lat (usec)   : 500=0.01%, 750=1.42%, 1000=12.40%
  lat (msec)   : 2=55.47%, 4=6.27%, 10=19.89%, 20=4.23%, 50=0.31%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.25%, sys=1.35%, ctx=978340, majf=0, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=630793,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=10.3MiB/s (10.8MB/s), 10.3MiB/s-10.3MiB/s (10.8MB/s-10.8MB/s), io=2464MiB (2584MB), run=240005-240005msec

Testing: Random write performance
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 8 processes
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=4256KiB/s][w=1064 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=8): err= 0: pid=140475: Sat Sep 28 09:20:48 2024
  write: IOPS=1329, BW=5320KiB/s (5447kB/s)(1247MiB/240076msec); 0 zone resets
    slat (usec): min=9, max=23259, avg=49.30, stdev=113.73
    clat (usec): min=987, max=270938, avg=5961.55, stdev=15564.56
     lat (usec): min=1003, max=270969, avg=6011.19, stdev=15564.97
    clat percentiles (usec):
     |  1.00th=[  1565],  5.00th=[  1778], 10.00th=[  1893], 20.00th=[  2057],
     | 30.00th=[  2180], 40.00th=[  2278], 50.00th=[  2376], 60.00th=[  2507],
     | 70.00th=[  2638], 80.00th=[  2933], 90.00th=[  6652], 95.00th=[ 19268],
     | 99.00th=[ 98042], 99.50th=[116917], 99.90th=[137364], 99.95th=[145753],
     | 99.99th=[214959]
   bw (  KiB/s): min= 1392, max= 9754, per=100.00%, avg=5325.46, stdev=176.57, samples=3832
   iops        : min=  348, max= 2438, avg=1331.23, stdev=44.13, samples=3832
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=16.19%, 4=70.71%, 10=4.70%, 20=3.53%, 50=2.07%
  lat (msec)   : 100=1.85%, 250=0.94%, 500=0.01%
  cpu          : usr=0.15%, sys=0.78%, ctx=465152, majf=0, minf=79
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,319282,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5320KiB/s (5447kB/s), 5320KiB/s-5320KiB/s (5447kB/s-5447kB/s), io=1247MiB (1308MB), run=240076-240076msec

Testing: IO Latency on individual request
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=1 time=1.54 ms (warmup)
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=2 time=1.98 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=3 time=1.67 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=4 time=1.71 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=5 time=1.64 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=6 time=1.63 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=7 time=1.63 ms (fast)
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=8 time=1.66 ms
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=9 time=1.54 ms (fast)
4 KiB <<< . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB): request=10 time=1.47 ms (fast)

--- . (ceph 10.2.214.1:6789,10.2.214.3:6789,10.2.214.4:6789:/volumes/csi/csi-vol-02fc1163-a67d-4cd3-8483-7b4545dacc19/edeaae84-b42c-4090-a814-61e2e0d9c10c 800 GiB) ioping statistics ---
9 requests completed in 14.9 ms, 36 KiB read, 602 iops, 2.36 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 1.47 ms / 1.66 ms / 1.98 ms / 132.2 us
jianwang-ntu commented 2 days ago

The tool can be found from https://gist.github.com/jsandas/a3705a619214470b2d92b1fb2b559ad9