search

rust-lang / crater

Run experiments across parts of the Rust ecosystem!
https://crater.rust-lang.org
643 stars 90 forks source link

Crater seems to sometimes test outdated crates #727

Open RalfJung opened 6 months ago

RalfJung commented 6 months ago

The crater run at https://crater-reports.s3.amazonaws.com/beta-1.78-1/index.html tested version 2.0.1 of the stabby-abi crate. However, at the time of that crater run (which unfortunately not visible in the report, you have to find the PR to see it started on March 26), version 4 of that crate had already been released (release of 4.0.0: March 4). Why did crater test an outdated version of the crate?

This unfortunately led to us missing a regression, since stabby 2 failed to build with the stable baseline for that beta regression run, making crater unable to detect further regressions. See here for context.

Cc @rust-lang/infra -- not sure who is maintaining crater?

RalfJung commented 6 months ago

In fact the latest beta run at https://crater-reports.s3.amazonaws.com/beta-1.79-1/full.html still tests stabby-abi 2.0.1. Something seems very wrong here.

RalfJung commented 5 months ago

@Mark-Simulacrum has manually done something that should fix this for now -- but I don't know if the underlying issue is something that can repeat.

Mark-Simulacrum commented 5 months ago

My fix is likely just a resync, underlying issue is definitely not fixed.

RalfJung commented 3 months ago

This seems to be happening again.