Fuuzetsu
commented
3 years ago Aha! Of course after posting an issue, I get further. I found that I can shove some code into create_resolver and gut other places a bit too to load rlibs during compilation:
pub fn create_resolver(
sess: Lrc<Session>,
metadata_loader: Box<MetadataLoaderDyn>,
krate: &ast::Crate,
crate_name: &str,
) -> BoxedResolver {
tracing::trace!("create_resolver for {}: {:?}", crate_name, krate);
if crate_name == "cloudsim" {
let dbg = |path: &'static str| {
let data = rustc_metadata::locator::get_metadata_section(&sess.target, rustc_metadata::locator::CrateFlavor::Rlib, &std::path::Path::new(path), &*metadata_loader).unwrap();
let mut buf: Vec<u8> = format!("XXX META FOR {}:\n", path).as_bytes().to_vec();
data.list_crate_metadata(&mut buf).unwrap();
let buf = String::from_utf8(buf).unwrap();
tracing::trace!("{}", buf);
};
dbg("/nix/store/pfy5c0x8jbx44xfnsrqb5yp5sz5ahpxx-rust_parse-display-0.5.2-lib/lib/libparse_display-38cc57346a.rlib");
dbg("/tmp/foo/libparse_display-38cc57346a.rlib");
}This gives us:
TRACE rustc_interface::passes XXX META FOR /nix/store/pfy5c0x8jbx44xfnsrqb5yp5sz5ahpxx-rust_parse-display-0.5.2-lib/lib/libparse_display-38cc57346a.rlib:
Crate info:
name parse_display-38cc57346a
hash 5f612fa7f4240092 stable_crate_id StableCrateId(16881853909076815645)
proc_macro false
=External Dependencies=
1 std-008055cc7d873802 hash a2dafebbcfe30fb2 host_hash None kind Explicit
2 core-4beb03d03503c439 hash f808ea40ffa97e9f host_hash None kind Explicit
3 compiler_builtins-dd7db1bec6909f24 hash d1ef612ea1c41b0b host_hash None kind Explicit
4 rustc_std_workspace_core-557ba8776e04d182 hash 5b57ea45fda7dad2 host_hash None kind Explicit
5 alloc-ac23a75f6f42004e hash a7f45db4bd465ca7 host_hash None kind Explicit
6 libc-8480e85e0be96197 hash d2acc6a19075b83f host_hash None kind Explicit
7 unwind-74be3a703f788ba2 hash f299ac5ad662548e host_hash None kind Explicit
8 cfg_if-8f2c5b445c28b2e3 hash 32ab094aa7b255b0 host_hash None kind Explicit
9 hashbrown-6b148909d375a785 hash cae2243f7ad96800 host_hash None kind Explicit
10 rustc_std_workspace_alloc-cd15fa647f4775d1 hash 2f5ca7c7bd62d0ed host_hash None kind Explicit
11 rustc_demangle-e530649c9a06e3c6 hash 78f9818ae04e0955 host_hash None kind Explicit
12 std_detect-0c35b278736219a2 hash 5b8a3aec28451004 host_hash None kind Explicit
13 addr2line-d489f0ca872880cc hash db8132e99d16ec5e host_hash None kind Explicit
14 gimli-75f07df0b18fea39 hash a7acca3f4fd97b7b host_hash None kind Explicit
15 object-95c14e1c1e3ebcc4 hash 22a1c9e36fdb00d2 host_hash None kind Explicit
16 miniz_oxide-f9a3c3274a1835e0 hash e41cb10aab3e61cc host_hash None kind Explicit
17 adler-d4cbb754ee9f4daa hash 0f4ca401300c8a8e host_hash None kind Explicit
18 panic_unwind-06f01ac2578bda94 hash 56aaadbcd17bec94 host_hash None kind Implicit
19 once_cell-5f618aa59f hash 8b5771286d8f05d2 host_hash None kind Explicit
20 regex-b01209c1e1 hash 3227fdded68e0222 host_hash None kind Explicit
21 regex_syntax-52823be8f8 hash de4fceec08d3120b host_hash None kind Explicit
22 aho_corasick-548c55c4a6 hash 2c940b422112251a host_hash None kind Explicit
23 memchr-131ec80d41 hash 0a80111741a8b8c7 host_hash None kind Explicit
24 parse_display_derive-c5a2af5848 hash 8b441a0395bccd4e host_hash None kind MacrosOnly
TRACE rustc_interface::passes XXX META FOR /tmp/foo/libparse_display-38cc57346a.rlib:
Crate info:
name parse_display-38cc57346a
hash 5963747a8c8f099e stable_crate_id StableCrateId(16881853909076815645)
proc_macro false
=External Dependencies=
1 std-008055cc7d873802 hash a2dafebbcfe30fb2 host_hash None kind Explicit
2 core-4beb03d03503c439 hash f808ea40ffa97e9f host_hash None kind Explicit
3 compiler_builtins-dd7db1bec6909f24 hash d1ef612ea1c41b0b host_hash None kind Explicit
4 rustc_std_workspace_core-557ba8776e04d182 hash 5b57ea45fda7dad2 host_hash None kind Explicit
5 alloc-ac23a75f6f42004e hash a7f45db4bd465ca7 host_hash None kind Explicit
6 libc-8480e85e0be96197 hash d2acc6a19075b83f host_hash None kind Explicit
7 unwind-74be3a703f788ba2 hash f299ac5ad662548e host_hash None kind Explicit
8 cfg_if-8f2c5b445c28b2e3 hash 32ab094aa7b255b0 host_hash None kind Explicit
9 hashbrown-6b148909d375a785 hash cae2243f7ad96800 host_hash None kind Explicit
10 rustc_std_workspace_alloc-cd15fa647f4775d1 hash 2f5ca7c7bd62d0ed host_hash None kind Explicit
11 rustc_demangle-e530649c9a06e3c6 hash 78f9818ae04e0955 host_hash None kind Explicit
12 std_detect-0c35b278736219a2 hash 5b8a3aec28451004 host_hash None kind Explicit
13 addr2line-d489f0ca872880cc hash db8132e99d16ec5e host_hash None kind Explicit
14 gimli-75f07df0b18fea39 hash a7acca3f4fd97b7b host_hash None kind Explicit
15 object-95c14e1c1e3ebcc4 hash 22a1c9e36fdb00d2 host_hash None kind Explicit
16 miniz_oxide-f9a3c3274a1835e0 hash e41cb10aab3e61cc host_hash None kind Explicit
17 adler-d4cbb754ee9f4daa hash 0f4ca401300c8a8e host_hash None kind Explicit
18 panic_unwind-06f01ac2578bda94 hash 56aaadbcd17bec94 host_hash None kind Implicit
19 once_cell-5f618aa59f hash 8b5771286d8f05d2 host_hash None kind Explicit
20 regex-b01209c1e1 hash 3227fdded68e0222 host_hash None kind Explicit
21 regex_syntax-52823be8f8 hash de4fceec08d3120b host_hash None kind Explicit
22 aho_corasick-548c55c4a6 hash 2c940b422112251a host_hash None kind Explicit
23 memchr-131ec80d41 hash 0a80111741a8b8c7 host_hash None kind Explicit
24 parse_display_derive-c5a2af5848 hash 1701d69e3497ed1f host_hash None kind MacrosOnlyThe line 24 is different: so the problem is the same just one level down. I will now check rlib for that and keep going until I find the one package that has a difference somewhere else. Though, macro package looks promising honestly and it might come up right there.
klensy
workingjubilee
We're using
nixto build rust crates and benefit from binary caches which are usually populated by our CI. I'll assume a slightly familiarity but I can elaborate when necessary. The details are not too important but it means that the following scenario can happen:Consider a crate
Cthat depends on crateBthat depends on crateA(that depends onstd). Now, the following sequence of events happens.Alocally. Let's call this build resultA-local. This goes into the/nix/storeandAis never rebuilt again (it is in immutable location) and is always reused when all the inputs (sources, dependencies) are identical.A,BandC. CI is also using nix and binary cache is currently empty.A: this has exactly the same inputs and so is given exactly same hash. Let's call itA-cache. It pushes pushesA-cacheto some binary cache (in S3 for example).Busing theA-cacheit just built, let's say it's calledB-cache. It pushes it to binary cache too.CasC-cacheand pushes to binary cache. Everything is great, CI is green.Con local machine, perhaps it is a binary crate . However, the developer first makes some changes toCsource code, maybe added some debug info. So what do we have to build?Adoesn't need building: it's already stored locally asA-localin local/nix/storeBis needed to buildC: luckilyBexists in a binary cache asB-cacheso we just download it.Csource code changed so we have to build it, we haveA-localandB-cachein the dependency tree now.Does the build succeed? Most of the time, yes. But sometimes, it fails if rustc produced two incompatible results for
A: that is, ifA-localandA-cachediffer, we're in trouble. The developer can't useA-localwithB-cache: onlyA-cacheis usable.An error message might look like:
In my real case,
Aisparse_displaycreate,Bis our local crate calledtraderandCwas another crate in the workspace that depended ontrader. AsB(trader) came from binary cache, it is binary-identical and we only have to focus onA.Investigation
First, let's see what rustc says:
Looks like the hash it expects is not what it gets. Back to this later. Let's try to look for something obvious in the
.rlib.If I compare binaries side-by-side, it looks like the two files are 1 byte misaligned in first section and have minor differences later on. I will include both files for your own inspection too.
Of course, it's very difficult to tell anything by just looking at raw binary. I tried to add extra information to
rustcitself so that I could see during the hashing what values it's getting: if hash inputs were different, we'd see which ones and be able to go backwards from there. Looking around, I thinkcrate_hashfunction is the point to changeSadly, this doesn't work for ironically similar reason: building this compiler locally gives a new
stdhash which means I can't re-invoke originalrustccommand with the original.rlibsas those depend onstdwhich has the wrong hash...At this point I decided to open this ticket: how can I debug this further? At this point I am more or less getting ready to try to gut
rustcto the point that I can callcrate_hashmyself or the rlibs directly but it seems like a lot of work and I'm not confident it's even going to work. I didn't find any tool to readrlibfiles either.If we can find what the difference between the two files it, presumably we can work backwards and find why
rustcproduced two different results.Here are the files: different_rlibs.tar.gz.
localwas one I built locally andcacheis from the binary cache, built by CI machine.Meta
rustc --version --verbose:I originally suspected that codegen-units may be the culprit: some concurrent resource getting non-deterministically modified, such as to hand out some Ids or something that made it to the code. But no, the issue occurred even with codegen-units=1.
It is difficult to verify or replicate the bug because it happens fairly rarely so it seems like looking at the rlibs I attached above is probably the fastest way to figure something out.