search

CodSpeedHQ / pytest-codspeed

Pytest plugin to create CodSpeed benchmarks
https://codspeed.io
MIT License
53 stars 6 forks source link

Issue on test comparison #17

Closed matroscoe closed 1 year ago

matroscoe commented 1 year ago

Sorry if this is not the correct repository to report this on but I figured a GitHub issue is easier to follow than a discord thread.

When I rebase I am not seeing the expected behavior where the feature branch has its benchmark it is being compared to reset to the latest value of in our case development

e.g. in my development branch this is the latest value that was merged in as seen from within codspeed: image

when I go to: https://codspeed.io/{org}/{github-repo}/benchmarks I see this: image

In my latest feature branch I see: image

my concern is that there is something different going on either in the benchmark comparison or in how the new "default" is run because all my feature branches show roughly the same performance drop which ties to the "latest benchmark" from https://codspeed.io/{org}/{github-repo}/benchmarks but I wouldn't expect to see so much variability from my feature branch to my development branch.

adriencaccia commented 1 year ago

Hey @matroscoe, thanks for raising the issue with such information!

It will be simpler to continue on Discord though, we just created a dedicated private channel and invited you to it. We are currently looking at the data to understand what happened.

See you on Discord