Sadly, we have plenty of false positives (Cc https://github.com/rust-lang/crater/issues/663 but there are other common problems, too, such as strange proc-macro trait failures). Besides the ever-going question of reducing those, I think it'd also be good to mitigate this somewhat: when a regression is detected, crater could re-run the failing job to ensure it still fails the 2nd time.
Sadly, we have plenty of false positives (Cc https://github.com/rust-lang/crater/issues/663 but there are other common problems, too, such as strange proc-macro trait failures). Besides the ever-going question of reducing those, I think it'd also be good to mitigate this somewhat: when a regression is detected, crater could re-run the failing job to ensure it still fails the 2nd time.