search

crate / crate-benchmarks

A collection of CrateDB benchmarks.
7 stars 3 forks source link

find_regressions' print_diffs' output is not in chronological order #236

Open jeeminso opened 2 years ago

jeeminso commented 2 years ago

from : https://github.com/crate/crate-alerts/issues/112

Key(concurrency=15, stmt='select min("adRevenue") from uservisits group by "cCode"', bulk_size=None, meta_name='group_by.toml')

  4.7.0-22881aac7 → 4.7.0-dbcc240e2

   16.4%   910.380 → 1059.795

  linregress slope: 5.043

from above output, I expect 22881aac7 to be older than dbcc240e2, but 22881aac7 is merged on oct 28th and dbcc240e2 on 20th.

jeeminso commented 2 years ago

Hi @mfussenegger, could this be a bug? My understanding is that this output shows that the regression is fixed or was just a fluctuation.

mfussenegger commented 2 years ago

Could be - not sure. The script relies on the order returned by CrateDB https://github.com/crate/crate-benchmarks/blob/2777b698e00bf2c902cae7e3bf773156c5b12fe2/find_regressions.py#L77-L84

Maybe having the version info as order criteria is a problem?

jeeminso commented 2 years ago

Thanks, is there a way to connect to : https://crate-production.aks1.westeurope.azure.cratedb.net or to jenkins-04.bregenz.a1.cr8.net ? It would be ideal if I could actually look at the benchmark data

mfussenegger commented 2 years ago

https://cr8.net/zeppelin/#/ is connected to https://crate-production.aks1.westeurope.azure.cratedb.net

You could run ad-hoc queries there. I think if you're in the VPN it should also be possible to connect, but not sure.