search

axboe / fio

Flexible I/O Tester
GNU General Public License v2.0
5.23k stars 1.26k forks source link

FIO Client-Server model - Need to get one consolidated result #610

Closed Gokul29 closed 6 years ago

Gokul29 commented 6 years ago

I am running a FIO client server model on 4 nodes. node=1 will be acting as a server & node=1,2,3,4 will be acting as a client. Below is the FIO command & configuration file that i used,

fio --client=host.list /root/fio.cfg

fio.cfg:

[global]
ioengine=libaio
iodepth=16
rw=randread
bs=4k
numjobs=4
group_reporting=1
time_based=1
runtime=120
significant_figures=10
[job1]
filename=/dev/sdb
filename=/dev/sdc

Below is my output: 

hostname=rhel7.5-4, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.6, flags=1
hostname=rhel7.5-3, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.6, flags=1
hostname=rhel7.5-2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.6, flags=1
hostname=rhel7.5-1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.6, flags=1
<rhel7.5-1> job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
<rhel7.5-1> ...
<rhel7.5-2> job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
<rhel7.5-2> ...
<rhel7.5-4> job1: (g=0): rw=randread, <rhel7.5-3> job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodept
h=16
ioengine=libaio, iodepth=16
<rhel7.5-4> ...
<rhel7.5-3> ...
<rhel7.5-1> Starting 4 processes
<rhel7.5-2> Starting 4 processes
<rhel7.5-4> Starting 4 processes
<rhel7.5-3> Starting 4 processes
job1: (groupid=0, jobs=4): err= 0: pid=4738: Tue Jun 12 05:58:37 2018
   read: IOPS=827732, BW=3390393011B/s (3390393011B/s)(397314992KiB/120001msec)
    slat (nsec): min=882, max=183315k, avg=3158.02, stdev=192561.83
    clat (nsec): min=1174, max=294006k, avg=73102.82, stdev=965021.92
     lat (usec): min=2, max=294090, avg=76.41, stdev=999.73
    clat percentiles (usec):
     |  1.00th=[   30],  5.00th=[   30], 10.00th=[   30], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   31], 60.00th=[   31],
     | 70.00th=[   31], 80.00th=[   32], 90.00th=[   32], 95.00th=[   35],
     | 99.00th=[   47], 99.50th=[   57], 99.90th=[14484], 99.95th=[18220],
     | 99.99th=[33424]
   bw (  KiB/s): min=  440, max=1016423, per=24.67%, avg=816792.54, stdev=340113.02, samples=958
   iops        : min=  110, max=254105, avg=204197.82, stdev=85028.12, samples=958
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=99.38%
  lat (usec)   : 100=0.16%, 250=0.05%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.13%, 20=0.16%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=15.38%, sys=27.38%, ctx=69839, majf=0, minf=217
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=99328748,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3390393011B/s (3390393011B/s), 3390393011B/s-3390393011B/s (3390393011B/s-3390393011B/s), io=397314992KiB (406850552kB), run=120001-120001msec

Disk stats (read/write):
  sdb: ios=25601/0, merge=0/0, ticks=37890/0, in_queue=37878, util=14.23%
  sdc: ios=25604/0, merge=0/0, ticks=32119/0, in_queue=32112, util=13.45%
<rhel7.5-1>
job1: (groupid=0, jobs=4): err= 0: pid=4506: Tue Jun 12 05:58:37 2018
   read: IOPS=777966, BW=3186549453B/s (3186549453B/s)(373426876KiB/120001msec)
    slat (nsec): min=959, max=234024k, avg=3305.95, stdev=195733.00
    clat (nsec): min=1405, max=324862k, avg=77821.59, stdev=943264.94
     lat (usec): min=2, max=324985, avg=81.27, stdev=973.74
    clat percentiles (usec):
     |  1.00th=[   32],  5.00th=[   32], 10.00th=[   33], 20.00th=[   33],
     | 30.00th=[   33], 40.00th=[   33], 50.00th=[   33], 60.00th=[   34],
     | 70.00th=[   34], 80.00th=[   34], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   50], 99.50th=[   70], 99.90th=[17957], 99.95th=[18220],
     | 99.99th=[30278]
   bw (  KiB/s): min=  944, max=1231419, per=24.65%, avg=767120.19, stdev=307336.29, samples=956
   iops        : min=  236, max=307854, avg=191779.73, stdev=76833.95, samples=956
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=99.18%
  lat (usec)   : 100=0.33%, 250=0.05%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.15%, 20=0.17%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=15.52%, sys=27.73%, ctx=70023, majf=0, minf=223
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=93356719,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3186549453B/s (3186549453B/s), 3186549453B/s-3186549453B/s (3186549453B/s-3186549453B/s), io=373426876KiB (382389121kB), run=120001-120001msec

Disk stats (read/write):
  sdb: ios=25600/0, merge=0/0, ticks=39099/0, in_queue=39086, util=13.64%
  sdc: ios=25602/0, merge=0/0, ticks=26531/0, in_queue=26520, util=12.20%
<rhel7.5-2>
job1: (groupid=0, jobs=4): err= 0: pid=4459: Tue Jun 12 05:58:37 2018
   read: IOPS=759505, BW=3110935568B/s (3110935568B/s)(364565800KiB/120001msec)
    slat (nsec): min=980, max=264384k, avg=3393.02, stdev=204338.84
    clat (nsec): min=1291, max=310618k, avg=79693.71, stdev=982208.29
     lat (usec): min=2, max=350862, avg=83.24, stdev=1014.86
    clat percentiles (usec):
     |  1.00th=[   33],  5.00th=[   33], 10.00th=[   33], 20.00th=[   34],
     | 30.00th=[   34], 40.00th=[   34], 50.00th=[   34], 60.00th=[   34],
     | 70.00th=[   34], 80.00th=[   35], 90.00th=[   35], 95.00th=[   44],
     | 99.00th=[   51], 99.50th=[  113], 99.90th=[17171], 99.95th=[18220],
     | 99.99th=[31065]
   bw (  KiB/s): min=  344, max=919405, per=24.67%, avg=749548.48, stdev=302558.98, samples=956
   iops        : min=   86, max=229851, avg=187386.79, stdev=75639.62, samples=956
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=98.71%
  lat (usec)   : 100=0.79%, 250=0.05%, 500=0.06%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.16%, 20=0.17%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=15.40%, sys=27.77%, ctx=70682, majf=0, minf=213
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=91141450,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3110935568B/s (3110935568B/s), 3110935568B/s-3110935568B/s (3110935568B/s-3110935568B/s), io=364565800KiB (373315379kB), run=120001-120001msec

Disk stats (read/write):
  sdb: ios=25601/0, merge=0/0, ticks=31295/0, in_queue=31285, util=12.91%
  sdc: ios=25608/0, merge=0/0, ticks=34928/0, in_queue=34908, util=13.39%
<rhel7.5-4>
job1: (groupid=0, jobs=4): err= 0: pid=4439: Tue Jun 12 05:58:37 2018
   read: IOPS=795893, BW=3259980871B/s (3259980871B/s)(382032192KiB/120001msec)
    slat (nsec): min=967, max=188583k, avg=3166.06, stdev=187698.95
    clat (nsec): min=1202, max=219913k, avg=76137.83, stdev=886131.59
     lat (usec): min=2, max=220039, avg=79.45, stdev=911.27
    clat percentiles (usec):
     |  1.00th=[   32],  5.00th=[   33], 10.00th=[   33], 20.00th=[   33],
     | 30.00th=[   34], 40.00th=[   34], 50.00th=[   34], 60.00th=[   34],
     | 70.00th=[   34], 80.00th=[   35], 90.00th=[   35], 95.00th=[   43],
     | 99.00th=[   50], 99.50th=[   73], 99.90th=[17171], 99.95th=[18220],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1304, max=1071936, per=24.65%, avg=784823.94, stdev=268318.73, samples=956
   iops        : min=  326, max=267984, avg=196205.66, stdev=67079.59, samples=956
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=99.13%
  lat (usec)   : 100=0.39%, 250=0.05%, 500=0.06%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.15%, 20=0.17%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=16.12%, sys=28.79%, ctx=70890, majf=0, minf=213
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=95508048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
All clients: (groupid=0, jobs=4): err= 0: pid=0: Tue Jun 12 05:58:37 2018
   read: IOPS=3161098, BW=12644393.5Ki (12947858.9k)(1517339860KiB/120001msec)
    slat (nsec): min=882, max=264384k, avg=3291.65, stdev=168318.76
    clat (nsec): min=1174, max=324862k, avg=77847.46, stdev=805276.60
     lat (usec): min=2, max=350862, avg=81.29, stdev=830.60
   bw (  KiB/s): min=  344, max=1231419, per=6.07%, avg=767153.18, stdev=254104.92, samples=3826
   iops        : min=   86, max=307854, avg=191787.97, stdev=63526.13, samples=3826
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=99.11%
  lat (usec)   : 100=0.41%, 250=0.05%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.15%, 20=0.17%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=15.61%, sys=27.92%, ctx=281434, majf=0, minf=866
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=379334965,0,0,0 short=0,0,0,0 dropped=0,0,0,0

Run status group 0 (all jobs):
   READ: bw=3259980871B/s (3259980871B/s), 3259980871B/s-3259980871B/s (3259980871B/s-3259980871B/s), io=382032192KiB (391200965kB), run=120001-120001msec

Disk stats (read/write):
  sdb: ios=25601/0, merge=0/0, ticks=24178/0, in_queue=24167, util=9.91%
  sdc: ios=25602/0, merge=0/0, ticks=24941/0, in_queue=24931, util=9.96%
----------------------------------------------------------------------------------------------------- **Queries:** 1. From the above results, i have getting each client output also i was getting output with **All clients** output. Is that **All Client** output is the consolidated result for all 4 nodes ? 2. If above is yes, why do we get the All client results in in-between the node result part? Is this misalignment in result file or this the standard format ? 3. Is there any fio command option to get only consolidated results in result file instead of all individual node results ? Thanks in advance.
sitsofe commented 6 years ago

As these aren't so much issues in fio and are more "is this correct?" questions they would be better aimed at the fio mailing list. I'll answer here this time but I'll be more strict in the future :-)

  1. No, each client's output is considered a separate group to the others (aggregation is actually done on the machines acting as servers). Relatedly, I don't know how you would you merge some of the information (e.g. consider disk stats where two machines may have a distinct /dev/sdc and the jobs sent used them both)...
  2. See above.
  3. Not at the moment - for now you have to postprocess fio's output with another tool...