search

rust-lang / cargo-bisect-rustc

Bisects rustc, either nightlies or CI artifacts
https://rust-lang.github.io/cargo-bisect-rustc/
Apache License 2.0
178 stars 55 forks source link

git bisecting broken? #134

Closed matthiaskrgr closed 3 years ago

matthiaskrgr commented 3 years ago

I was trying to bisect https://github.com/rust-lang/rust/issues/87308 with --start=2021-07-18 --end=2021-07-20

While bisecting the actual nightly seemed to have worked, cargo-bisect-rustc failed to bisect the actual PR

installing nightly-2021-07-18
cargo for x86_64-unknown-linux-gnu: 6.07 MB / 6.07 MB [=================================================================================================================================================================================================================================================] 100.00 % 6.77 MB/s testing...
RESULT: nightly-2021-07-18, ===> No
uninstalling nightly-2021-07-18

installing nightly-2021-07-20
cargo for x86_64-unknown-linux-gnu: 6.07 MB / 6.07 MB [=================================================================================================================================================================================================================================================] 100.00 % 6.76 MB/s testing...
RESULT: nightly-2021-07-20, ===> Yes
uninstalling nightly-2021-07-20

installing nightly-2021-07-19
testing...
RESULT: nightly-2021-07-19, ===> Yes
uninstalling nightly-2021-07-19

searched toolchains nightly-2021-07-18 through nightly-2021-07-20

********************************************************************************
Regression in nightly-2021-07-19
********************************************************************************

fetching https://static.rust-lang.org/dist/2021-07-18/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2021-07-18: 40 B / 40 B [============================================================================================================================================================================================================================================================] 100.00 % 640.76 KB/s converted 2021-07-18 to c7331d65bdbab1187f5a9b8f5b918248678ebdb9
fetching https://static.rust-lang.org/dist/2021-07-19/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2021-07-19: 40 B / 40 B [============================================================================================================================================================================================================================================================] 100.00 % 256.19 KB/s converted 2021-07-19 to 59216858a323978a97593cba22b5ed84350a3783
looking for regression commit between 2021-07-18 and 2021-07-19
cloning rust repository
fetching (via local git) commits from c7331d65bdbab1187f5a9b8f5b918248678ebdb9 to 59216858a323978a97593cba22b5ed84350a3783
opening existing repository at "rust.git"
refreshing repository
looking up first commit
looking up second commit
checking that commits are by bors and thus have ci artifacts...
finding bors merge commits
found 8 bors merge commits in the specified range
  commit[0] 2021-07-17UTC: Auto merge of #87203 - jackh726:logging, r=nikomatsakis
  commit[1] 2021-07-17UTC: Auto merge of #87129 - FabianWolff:issue-75356, r=varkor
  commit[2] 2021-07-18UTC: Auto merge of #85686 - ptrojahn:loop_reinitialize, r=estebank
  commit[3] 2021-07-18UTC: Auto merge of #87071 - inquisitivecrystal:inclusive-range, r=estebank
  commit[4] 2021-07-18UTC: Auto merge of #87242 - JohnTitor:rollup-t9rmwpo, r=JohnTitor
  commit[5] 2021-07-18UTC: Auto merge of #86698 - cjgillot:modc, r=estebank
  commit[6] 2021-07-18UTC: Auto merge of #87252 - RalfJung:miri, r=RalfJung
  commit[7] 2021-07-18UTC: Auto merge of #86950 - tmiasko:personality, r=nagisa
validated commits found, specifying toolchains

installing c7331d65bdbab1187f5a9b8f5b918248678ebdb9
cargo for x86_64-unknown-linux-gnu: 6.07 MB / 6.07 MB [=================================================================================================================================================================================================================================================] 100.00 % 3.42 MB/s testing...
RESULT: c7331d65bdbab1187f5a9b8f5b918248678ebdb9, ===> No
uninstalling c7331d65bdbab1187f5a9b8f5b918248678ebdb9

installing 59216858a323978a97593cba22b5ed84350a3783
cargo for x86_64-unknown-linux-gnu: 6.07 MB / 6.07 MB [=================================================================================================================================================================================================================================================] 100.00 % 3.38 MB/s testing...
RESULT: 59216858a323978a97593cba22b5ed84350a3783, ===> No
uninstalling 59216858a323978a97593cba22b5ed84350a3783

ERROR: the commit at the end of the range (59216858a323978a97593cba22b5ed84350a3783) does not reproduce the regression
ehuss commented 3 years ago

Hm, something seems strange here. When I run the test, I get that nightly-2021-07-19 does not regress. Did you use --regress=ice? Perhaps run with -vv to see why it thinks nightly-2021-07-19 failed?

Successful build with 2021-07-19:

> RUSTFLAGS=-Zunpretty=everybody_loops cargo +bisector-nightly-2021-07-19-x86_64-apple-darwin build
   Compiling z72 v0.1.0 (/Users/eric/Proj/rust/cargo/scratch/z72)
#![feature(prelude_import)]
#[prelude_import]
use std::prelude::rust_2018::*;
#[macro_use]
extern crate std;
macro_rules! foo {
    () => { break 'x ; } //~ ERROR use of undeclared label `'x`
}

pub fn main() { loop { } }
    Finished dev [unoptimized + debuginfo] target(s) in 0.16s
dtolnay commented 3 years ago

I've hit this too. I think it is related to #123. It's off-by-one somewhere.

For example in https://github.com/rust-lang/rust/issues/88043, where cargo +nightly-2021-08-14 build is successful while cargo +nightly-2021-08-15 build is regressed, cargo-bisect-rustc has this unexpected behavior:

$ cargo-bisect-rustc --start 2021-08-14 --end 2021-08-15 -- build
checking nightly-2021-08-14 from the currently installed default nightly toolchain as the last failure
installing nightly-2021-08-14
testing...
RESULT: nightly-2021-08-14, ===> Yes
uninstalling nightly-2021-08-14

ERROR: the start of the range (nightly-2021-08-14) must not reproduce the regression

Okay, let's try to offset by one. But it still wrongly concludes the regression is in nightly-2021-08-14 (whereas it's really in nightly-2021-08-15) and then predictably fails to bisect at the commit level.

$ cargo-bisect-rustc --start 2021-08-13 --end 2021-08-15 -- build
installing nightly-2021-08-13
RESULT: nightly-2021-08-13, ===> No
uninstalling nightly-2021-08-13

installing nightly-2021-08-15
RESULT: nightly-2021-08-15, ===> Yes
uninstalling nightly-2021-08-15

installing nightly-2021-08-14
testing...
RESULT: nightly-2021-08-14, ===> Yes
uninstalling nightly-2021-08-14

searched toolchains nightly-2021-08-13 through nightly-2021-08-15

********************************************************************************
Regression in nightly-2021-08-14
********************************************************************************

fetching https://static.rust-lang.org/dist/2021-08-13/channel-rust-nightly-git-commit-hash.txt
fetching https://static.rust-lang.org/dist/2021-08-14/channel-rust-nightly-git-commit-hash.txt
looking for regression commit between 2021-08-13 and 2021-08-14
opening existing repository at "/git/bisect.rustc"
refreshing repository
fetching (via local git) commits from 0fa3190394475a84360b34e074e719d519bc40f1 to 5a19ffe1c2b99d9e09706cc286aad1ec0868eddb
opening existing repository at "/git/bisect.rustc"
refreshing repository
looking up first commit
looking up second commit
checking that commits are by bors and thus have ci artifacts...
finding bors merge commits
found 8 bors merge commits in the specified range
  commit[0] 2021-08-12UTC: Auto merge of #87916 - nbdd0121:black_box, r=nagisa
  commit[1] 2021-08-12UTC: Auto merge of #87980 - Manishearth:rollup-vkuix3y, r=Manishearth
  commit[2] 2021-08-13UTC: Auto merge of #87927 - spastorino:use-def-id-typeckresults, r=oli-obk
  commit[3] 2021-08-13UTC: Auto merge of #87954 - flip1995:clippyup, r=Manishearth
  commit[4] 2021-08-13UTC: Auto merge of #87956 - m-ou-se:closure-migration-macro-body, r=Aaron1011
  commit[5] 2021-08-13UTC: Auto merge of #88009 - GuillaumeGomez:rollup-f194yyk, r=GuillaumeGomez
  commit[6] 2021-08-13UTC: Auto merge of #87984 - m-ou-se:closure-lint-wording, r=Aaron1011
  commit[7] 2021-08-13UTC: Auto merge of #86492 - hyd-dev:no-mangle-method, r=petrochenkov
validated commits found, specifying toolchains

installing 0fa3190394475a84360b34e074e719d519bc40f1
RESULT: 0fa3190394475a84360b34e074e719d519bc40f1, ===> No
uninstalling 0fa3190394475a84360b34e074e719d519bc40f1

installing 5a19ffe1c2b99d9e09706cc286aad1ec0868eddb
RESULT: 5a19ffe1c2b99d9e09706cc286aad1ec0868eddb, ===> No
uninstalling 5a19ffe1c2b99d9e09706cc286aad1ec0868eddb

ERROR: the commit at the end of the range (5a19ffe1c2b99d9e09706cc286aad1ec0868eddb) does not reproduce the regression
ehuss commented 3 years ago

Oh! I have a guess as to what is wrong. Which version of cargo-bisect-rustc are you using? I always use latest master (actually, I use a local branch with several other fixes). The last published release was a long time ago, and there was an off-by-one fix done in the meantime (#113), among many other fixes. Is that maybe the case here?

camelid commented 3 years ago

I think I was using the latest released version, 0.6.0, in #123.

ehuss commented 3 years ago

@spastorino Would you be able to publish a new version? Or, if you would like people to help, I can publish if you invite me (though I'm not sure who has write access here). It would also be helpful if someone could take a look at #130 and #131, as the current master has issues for me.

spastorino commented 3 years ago

@ehuss sorry that I've missed your message. I've added you on crates.io. Let me know if you're able to publish.

ehuss commented 3 years ago

I have access on crates.io now, but I'll need to be able to push to this repo to update the version and such. @Mark-Simulacrum would it be alright to give me write access to this repo?

Mark-Simulacrum commented 3 years ago

Yeah, that makes sense to me. We should have some discussion about where the maintenance of this crate/repository best connects -- release team may make sense...

ehuss commented 3 years ago

I just published 0.6.1. I'm going to close as I'm going to assume this was caused by #113 or maybe #125. Feel free to open a new issue if things don't seem resolved with the new version.