search

crate / crate-benchmarks

A collection of CrateDB benchmarks.
8 stars 4 forks source link

Add metrics extracted from JFR to compare_run #208

Closed mfussenegger closed 4 years ago

mfussenegger commented 4 years ago

Example output:

Q: select count(distinct "searchWord") from uservisits group by "cCode"
C: 5
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      945.619 ±  285.149 |    762.244 |    881.228 |    936.162 |   2187.417 |
|   V2    |      946.812 ±  160.640 |    722.209 |    916.833 |    958.813 |   1720.607 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +   0.13%                           +   3.96%
There is a 2.90% probability that the observed difference is not random, and the best estimate of that difference is 0.13%
The test has no statistical significance

System/JVM Metrics (durations in ms, byte-values in MB)
    |    YOUNG GC            |       OLD GC           |      HEAP         |     ALLOC
    |  cnt      avg      max |  cnt      avg      max |  initial     used |     rate      total
 V1 |   36    29.18    61.78 |    0     0.00     0.00 |     8590     2616 |  2666.24     132592
 V2 |   36    27.36    52.15 |    0     0.00     0.00 |     8590     2350 |  2445.92     132461

V1 top allocation frames
  BytesRef.utf8ToString():63420459696
  StringUTF16.compress(...):29273199552
  Float.valueOf(float):13816682747
  GroupingCollector.evalKeyInputs(List):9566020490
  ArrayList.<init>(int):9066434792
V2 top allocation frames
  BytesRef.utf8ToString():63354973784
  StringUTF16.compress(...):29735760496
  Float.valueOf(float):13807254735
  GroupingCollector.evalKeyInputs(List):9268620255
  ArrayList.<init>(int):9003726512