If the old compiler successfully built a crate, but failed to build its tests, and the new compiler failed to build the main crate, that should be reported as a regression (and vice-versa as a fix).
But both failing to build the main crate and failure to build the tests currently result in a BuildFail status (TestFail only comes from failure to run the tests):
Noticed in rust-lang/rust#80953 (in that case, the new compiler failed the same way as the old one, but if the new compiler did have the regression, it wouldn't have been noticed due to this).
If the old compiler successfully built a crate, but failed to build its tests, and the new compiler failed to build the main crate, that should be reported as a regression (and vice-versa as a fix).
But both failing to build the main crate and failure to build the tests currently result in a
BuildFailstatus (TestFailonly comes from failure to run the tests):https://github.com/rust-lang/crater/blob/674e4c2018d0ac9eab228986826d4ed35b9d988a/src/runner/test.rs#L230-L280
Noticed in rust-lang/rust#80953 (in that case, the new compiler failed the same way as the old one, but if the new compiler did have the regression, it wouldn't have been noticed due to this).