perf-regression: collect report at each staging release

aldettinger commented 2 years ago

The idea is to start collecting performance regression report at each staging release. From there, we would learn whether:

[ ] the report generation can be run on distinct machines, operating systems
[ ] reports looks consistent across versions, machines, operating systems
[x] the default duration of 10 minutes is a good trade-off to quickly spot potential regressions
[x] the current scenario provide the best value
[x] it would be valuable to reject a release with -5% regression detected

aldettinger commented 2 years ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.11.0  |       10m  |  15334.07 req/s [+0.00%]  |    8850.31 req/s [+0.00%]  |      OK  |
                2.12.0  |       10m  |  15343.82 req/s [+0.06%]  |    8976.51 req/s [+1.43%]  |      OK  |

zbendhiba commented 2 years ago

Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.12.0  |       10m  |  21492.67 req/s [+0.00%]  |   11516.03 req/s [+0.00%]  |      OK  |
                2.13.0  |       10m  |  21299.82 req/s [-0.90%]  |   11828.55 req/s [+2.71%]  |      OK  |

aldettinger commented 1 year ago

Camel Quarkus Throughput Performance Increase Compared to Previous Version                
Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.13.0  |       10m  |  38872.21 req/s [+0.00%]  |   22154.66 req/s [+0.00%]  |      OK  |
                2.15.0  |       10m  |  38605.46 req/s [-0.69%]  |   22438.48 req/s [+1.28%]  |      OK  |

aldettinger commented 1 year ago

 Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.15.0  |       10m  |  38922.37 req/s [+0.00%]  |   22605.97 req/s [+0.00%]  |      OK  |
                2.16.0  |       10m  |  39248.04 req/s [+0.84%]  |   22543.42 req/s [-0.28%]  |      OK  |

aldettinger commented 1 year ago

 Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.13.1  |       10m  |  38888.74 req/s [+0.00%]  |   22779.41 req/s [+0.00%]  |      OK  |
                2.13.2  |       10m  |  38918.29 req/s [+0.08%]  |   23068.36 req/s [+1.27%]  |      OK  |

zbendhiba commented 1 year ago

 Camel Quarkus Throughput Performance Increase Compared to Previous Version                             
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |               Status               |
------------------------------------------------------------------------------------------------------------------------------------
                2.16.0  |       10m  |  23394.75 req/s [+0.00%]  |   13429.86 req/s [+0.00%]  |                                OK  |
              3.0.0-M1  |       10m  |  23976.32 req/s [+2.49%]  |   12576.25 req/s [-6.36%]  |  Potential performance regression  |
2023-03-20 20:33:52,502 INFO  [io.quarkus] (main) perf-regression stopped in 0.006s

aldettinger commented 1 year ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.13.2  |       10m  |  39515.19 req/s [+0.00%]  |   23098.37 req/s [+0.00%]  |      OK  |
                2.13.3  |       10m  |  39745.16 req/s [+0.58%]  |   22774.74 req/s [-1.40%]  |      OK  |

ppalaga commented 1 year ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
              3.0.0-M1  |       10m  |  25477.00 req/s [+0.00%]  |   13664.18 req/s [+0.00%]  |      OK  |
              3.0.0-M2  |       10m  |  27129.07 req/s [+6.48%]  |   14002.18 req/s [+2.47%]  |      OK  |

zbendhiba commented 1 year ago

Comparison between version 2.16 and the latest 3.0.0-M2 to assess improvements since the M1 regression

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.16.0  |       10m  |  22896.22 req/s [+0.00%]  |   13133.90 req/s [+0.00%]  |      OK  |
              3.0.0-M2  |       10m  |  23067.02 req/s [+0.75%]  |   13404.82 req/s [+2.06%]  |      OK  |

ppalaga commented 1 year ago

               Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |  JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
---------------------------------------------------------------------------------------------------------
              3.0.0-M2  |       10m  |  9569.35 req/s [+0.00%]  |    6022.07 req/s [+0.00%]  |      OK  |
             3.0.0-RC1  |       10m  |  9091.22 req/s [-5.00%]  |    6312.80 req/s [+4.83%]  |      OK  |

aldettinger commented 1 year ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
             3.0.0-RC1  |       10m  |  39443.07 req/s [+0.00%]  |   22666.68 req/s [+0.00%]  |      OK  |
             3.0.0-RC2  |       10m  |  39347.05 req/s [-0.24%]  |   22559.33 req/s [-0.47%]  |      OK  |

aldettinger commented 1 year ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
             3.0.0-RC2  |       10m  |  40384.40 req/s [+0.00%]  |   22582.53 req/s [+0.00%]  |      OK  |
                 3.2.0  |       10m  |  39638.48 req/s [-1.85%]  |   22462.29 req/s [-0.53%]  |      OK  |

aldettinger commented 1 year ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.0  |       10m  |  39873.77 req/s [+0.00%]  |   22604.50 req/s [+0.00%]  |      OK  |
                 3.4.0  |       10m  |  39788.82 req/s [-0.21%]  |   22057.67 req/s [-2.42%]  |      OK  |

ppalaga commented 12 months ago

 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.0  |       10m  |  19495.67 req/s [+0.00%]  |   11577.53 req/s [+0.00%]  |      OK  |
                 3.2.1  |       10m  |  19257.06 req/s [-1.22%]  |   11733.54 req/s [+1.35%]  |      OK  |

jamesnetherton commented 11 months ago

                             Camel Quarkus Throughput Performance Increase Compared to Previous Version                              
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |               Status               |
-------------------------------------------------------------------------------------------------------------------------------------
                 3.4.0  |       10m  |   20250.91 req/s [+0.00%]  |   12564.35 req/s [+0.00%]  |                                OK  |
                 3.5.0  |       10m  |  15222.80 req/s [-24.83%]  |   9273.02 req/s [-26.20%]  |  Potential performance regression  |

aldettinger commented 11 months ago

                             Camel Quarkus Throughput Performance Increase Compared to Previous Version                              
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |               Status               |
-------------------------------------------------------------------------------------------------------------------------------------
                 3.4.0  |       10m  |   20250.91 req/s [+0.00%]  |   12564.35 req/s [+0.00%]  |                                OK  |
                 3.5.0  |       10m  |  15222.80 req/s [-24.83%]  |   9273.02 req/s [-26.20%]  |  Potential performance regression  |

Woops, this looks pretty bad. And it's confirmed on another machine, with 10m and 1h long runs. Let's deal with this in #5452.

jamesnetherton commented 11 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.1  |       10m  |  19241.25 req/s [+0.00%]  |   12220.36 req/s [+0.00%]  |      OK  |
                 3.2.2  |       10m  |  19023.79 req/s [-1.13%]  |   11801.31 req/s [-3.43%]  |      OK  |

jamesnetherton commented 10 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |  Status  |
-----------------------------------------------------------------------------------------------------------
                 3.5.0  |       10m  |   10184.32 req/s [+0.00%]  |    6655.12 req/s [+0.00%]  |      OK  |
                 3.6.0  |       10m  |  11328.00 req/s [+11.23%]  |   7618.24 req/s [+14.47%]  |      OK  |

zhfeng commented 10 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.2  |       10m  |  26745.20 req/s [+0.00%]  |   16353.16 req/s [+0.00%]  |      OK  |
                 3.2.3  |       10m  |  25663.29 req/s [-4.05%]  |   15666.90 req/s [-4.20%]  |      OK  |

jamesnetherton commented 8 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.6.0  |       10m  |  20520.59 req/s [+0.00%]  |   11885.86 req/s [+0.00%]  |      OK  |
                 3.7.0  |       10m  |  20577.29 req/s [+0.28%]  |   11885.30 req/s [-0.00%]  |      OK  |

aldettinger commented 7 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.7.0  |       10m  |  33532.36 req/s [+0.00%]  |   17749.33 req/s [+0.00%]  |      OK  |
                 3.8.0  |       10m  |  32340.26 req/s [-3.56%]  |   17498.96 req/s [-1.41%]  |      OK  |

jamesnetherton commented 6 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.0  |       10m  |  19217.32 req/s [+0.00%]  |   11275.71 req/s [+0.00%]  |      OK  |
                 3.8.1  |       10m  |  19457.29 req/s [+1.25%]  |   11112.60 req/s [-1.45%]  |      OK  |

aldettinger commented 6 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.0  |       10m  |  32583.61 req/s [+0.00%]  |   17899.62 req/s [+0.00%]  |      OK  |
                 3.9.0  |       10m  |  32610.64 req/s [+0.08%]  |   17575.19 req/s [-1.81%]  |      OK  |

jamesnetherton commented 5 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.9.0  |       10m  |  19907.84 req/s [+0.00%]  |   11237.27 req/s [+0.00%]  |      OK  |
                3.10.0  |       10m  |  19803.77 req/s [-0.52%]  |   11447.46 req/s [+1.87%]  |      OK  |

jamesnetherton commented 5 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.1  |       10m  |  19543.29 req/s [+0.00%]  |   11300.57 req/s [+0.00%]  |      OK  |
                 3.8.2  |       10m  |  19289.10 req/s [-1.30%]  |   11163.94 req/s [-1.21%]  |      OK  |

aldettinger commented 3 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.10.0  |       10m  |  31382.21 req/s [+0.00%]  |   16818.13 req/s [+0.00%]  |      OK  |
                3.11.0  |       10m  |  30932.51 req/s [-1.43%]  |   17040.53 req/s [+1.32%]  |      OK  |

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.11.0  |       10m  |  31612.49 req/s [+0.00%]  |   16867.78 req/s [+0.00%]  |      OK  |
                3.12.0  |       10m  |  30904.69 req/s [-2.24%]  |   17089.98 req/s [+1.32%]  |      OK  |

jamesnetherton commented 3 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.2  |       10m  |  19512.99 req/s [+0.00%]  |   11358.99 req/s [+0.00%]  |      OK  |
                 3.8.3  |       10m  |  20256.12 req/s [+3.81%]  |   11189.47 req/s [-1.49%]  |      OK  |

jamesnetherton commented 2 months ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.12.0  |       10m  |  19220.23 req/s [+0.00%]  |   11335.07 req/s [+0.00%]  |      OK  |
                3.13.0  |       10m  |  19540.48 req/s [+1.67%]  |   11387.82 req/s [+0.47%]  |      OK  |

aldettinger commented 1 month ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.13.0  |       10m  |  31313.87 req/s [+0.00%]  |   17044.78 req/s [+0.00%]  |      OK  |
                3.14.0  |       10m  |  31667.66 req/s [+1.13%]  |   17015.65 req/s [-0.17%]  |      OK  |

jamesnetherton commented 1 week ago

                Camel Quarkus Throughput Performance Increase Compared to Previous Version
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.14.0  |       10m  |  19539.85 req/s [+0.00%]  |   11221.73 req/s [+0.00%]  |      OK  |
                3.15.0  |       10m  |  19359.64 req/s [-0.92%]  |   10901.39 req/s [-2.85%]  |      OK  |

apache / camel-quarkus

perf-regression: collect report at each staging release #4012