rust-lang / crater

Run experiments across parts of the Rust ecosystem!
https://crater.rust-lang.org
627 stars 88 forks source link

"root results" does not list the biggest root regression #714

Open RalfJung opened 9 months ago

RalfJung commented 9 months ago

In https://crater-reports.s3.amazonaws.com/pr-117905/index.html, the biggest root regression is the bnum crate. This is easily verified by opening the "regressed: dependencies" category, it even explicitly lists "bnum-0.8.0 (249)" as a block of crates that failed because they depend on bnum.

However, "regressed: root results" does not list "bnum".

I was assuming that "root results" is the only category I have to go through to find out which are the root regressions, but it seems like that is not the case. Either there was a bug in looking for root regressions when generating the report, or the results page is labeled in very confusing ways.

Jules-Bertholet commented 9 months ago

What I think is going on here, is that bnum version 0.8.0 is regressed, but not the latest version 0.10. Crater only explicitly cargo builds the latest version of a crate. So when building bnum explicitly, Crater didn't encounter any errors, and therefore it is not listed as a root regression. But, in the process of building other crates that depend on old versions of bnum, errors were encountered building said old versions, so bnum is listed as a regressed dependency.

RalfJung commented 9 months ago

At the time of the run, the latest version was 0.9.1. It should also still have the regression.