OxDeadmeat / blktrace_merge

Merge blktrace events into a unified/singular io context
GNU General Public License v3.0
0 stars 0 forks source link

[RFE] Add Tukey boxplot data analysis for Q2C, Q2D and D2C values. #3

Open OxDeadmeat opened 1 week ago

OxDeadmeat commented 1 week ago

Is your feature request related to a problem? Please describe. A statistical analysis of the kernel vs storage wait times (Q2D vs D2C) would be appreciated. This would not be graphical in nature, that is no box and whisker plot but rather raw data output showing core Q2/3 data along with identification of outliers.

Describe the solution you'd like

A summary of Tukey data per device, per Q2D. D2C and Q2C -- something like the following. In this case it was determined that for device sdih, outliers started at 60.78ms up to the maximum observed Q2C of 345.38ms.

boxplot Device   min outliers  adj-min |<-  25%    50%    75% ->|  adj-max outliers          max : adj+body     mean    range      scsi-addr
boxplot sdih    0.44     0.00     0.00     0.91     3.32     6.18    14.08    60.78       345.38 :     3.96     7.94     5.27   ; 3:0:0:115 
boxplot sdhw    0.51     0.00     0.00     1.92     2.99     4.70     8.86    73.64       304.01 :     3.21     6.03     2.77   ; 3:0:0:94 

And additional, expanded information per device per Q2D, D2C and Q2C information similar to the following which includes breakdown into common buckets of percentile similar to fio along with median, average, and standard deviation values. In the following at the 99th percentile IO is completing in 3.67ms per io on average and there were some 2,375 io present in this bucket (99.00-99.49). So 99% of all io are completing (Q2C) in under 3.67ms.

boxplot Part             min outliers  adj-min |<-  25%    50%    75% ->|  adj-max outliers          max : adj+body     mean         r
boxplot sdaa            0.40     0.00     0.22     1.31     1.65     2.04     3.13     4.09         5.23 :     1.70     1.75     0.73 (values) iostat-sar01-06-  ; 1:0:0:36
boxplot sdaa                     0.00    24.00             50.00             24.00     2.00              :    98.00                   (percentage of part )
boxplot sdaa            0.00              0.10    25.00    50.00    75.00    98.00   100.00         4.09 :     0.00                   (percentage of total)
#_
#_        await (msec): min=  0.40 max=  5.23 avg=  1.75 mid=  0.00 bdy=  1.66 adj+bdy:  1.70 out=  3.55 r=  0.00 stddev=  0.00 scale=  0.00
#_        await percentiles (msec): idle 0 of 141 (%  0.00)
#_  |  1.00th=[  0.64|     592],  5.00th=[  0.88|    2375], 10.00th=[  1.00|    2969], 20.00th=[  1.22|    5938], 25.00th=[  1.31|    2968], 
#_  | 30.00th=[  1.42|    2969], 40.00th=[  1.53|    5938], 50.00th=[  1.65|    5938], 60.00th=[  1.79|    5937], 68.00th=[  1.96|    4750], 
#_  | 70.00th=[  1.99|    1188], 75.00th=[  2.04|    2969], 80.00th=[  2.15|    2968], 90.00th=[  2.50|    5938], 95.00th=[  2.85|    2969], 
#_  | 99.00th=[  3.67|    2375], 99.50th=[  4.09|     297], 99.70th=[  4.59|     118], 99.90th=[  5.23|     119], 99.95th=[  5.23|      30], 
#_  | 99.99th=[  5.23|      24], 101.00th=[  5.23|       6], 
#_  |                    hi adj  98.00th=[  3.55| 4249208]                hi outliers  58187.00th=[100.00| 6358272]

Describe alternatives you've considered A separate binary data file to be input to a new boxplot program could be implemented although the preference is to have the boxplot analysis to be embedded within the blktrace_merge tool itself.

Additional context none.