search

rust-lang / cargo-bisect-rustc

Bisects rustc, either nightlies or CI artifacts
https://rust-lang.github.io/cargo-bisect-rustc/
Apache License 2.0
178 stars 55 forks source link

bisect doesn't actually do bisection #136

Closed savente93 closed 3 years ago

savente93 commented 3 years ago

So please correct me if I'm wrong, but from what I understand looking at this code, cargo-bisect-rustc doesn't actually do bisection but instead linear backup? https://github.com/rust-lang/cargo-bisect-rustc/blob/67bc835fa54958ec024017e84379bc068a2fa458/src/main.rs#L821-L830

This isn't an issue if you're bisecting recent versions, but I'm looking into bisecting some regressions that were opened up in 2018 2020. (e.g. https://github.com/rust-lang/rust/issues/79168), and I saw at least one other issue (#76) mentioning this being a bit of a pain. Would a PR doing actual bisection here be likely to get merged? If so I'd be happy to work on one, but wanted to ask before hand. I'm thinking that if --end isn't specified, we can simply use 2015-10-20 (e.g. when std packages became available as stated in the comments.)

I can see that this is a bit unnecessary for folks bisecting more recent versions as mentioned, so perhaps we could keep the current code for them behind a --linear flag of some sort? Or if the current behaviour should remain the default maybe we could have a --bisect option of some description? Thoughts? Comments?

Mark-Simulacrum commented 3 years ago

Yeah, we used to do bisection, but in practice most of the time the regression is recent but you don't know a good --end flag. Plus, bisection relies on the script having a monotone function (just one change on the range), which in practice is often untrue -- the program usually has a MSRV of Rust 1.43 or something (i.e., not way back in 2015). So a straightforward bisection is likely to get confused because it will see failures with old nightlies which are false positives.

savente93 commented 3 years ago

the regression not being monotone is definitely something I hadn't thought about. Upon reflection, regressions that go back much father is less likely than I initially thought, so I'll close for now. Thanks for your feedback.

ehuss commented 3 years ago

Maybe I'm a bit confused by the question, but if you specify both--start and --end, then it will bisect between those two ranges. I use it quite frequently for very old bisections. As Mark says, you have to be careful about monotone behavior, but I've only run into that a few times.

rust-lang/rust#79168 looks like someone already bisected it to a specific nightly. In-between nightlies are only available for a short time (167 days I think?). Usually a culprit can be found with git log between those two commits. For very old bisects, dates should work. I also sometimes manually bisect releases (+1.15.0 then +1.30.0, and then in the middle, etc.) to narrow the range a little.