search

mozilla / sccache

Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
Apache License 2.0
5.84k stars 550 forks source link

[dist] failure when build.rs or procmacro depend on externalities #676

Open drahnr opened 4 years ago

drahnr commented 4 years ago

Having a workspace tree:

sub1/lvl2/Cargo.toml #depends on sub0 { version = .., path = ../../sub0 }
sub0/Cargo.toml

causes build failures where the Cargo.toml in sub0 can not be found

error: `/home/bernhard/Projects/substrate/primitives/wasm-interface/Cargo.toml` does not exist.
  --> /home/bernhard/Projects/substrate/primitives/wasm-interface/src/lib.rs:73:56
   |
73 | #[derive(PartialEq, Debug, Clone, Copy, codec::Encode, codec::Decode)]
   |                                                        ^^^^^^^^^^^^^

this assumes #666 is already applied / #87 is resolved to resolve to correct rustc versions, but is also present in latest master with a simpler setup.

drahnr commented 4 years ago

Again , compiling https://github.com/paritytech/substrate with cargo b

# <snip>
   Compiling trie-db v0.20.0
   Compiling frame-support-procedural v2.0.0-alpha.3 (/home/bernhard/Projects/substrate-perf-4876/frame/support/procedural)
   Compiling parity-scale-codec v1.2.0
   Compiling wasm-bindgen-futures v0.4.8
   Compiling impl-codec v0.4.2
   Compiling sp-wasm-interface v2.0.0-alpha.3 (/home/bernhard/Projects/substrate/primitives/wasm-interface)
error: `/home/bernhard/Projects/substrate/primitives/wasm-interface/Cargo.toml` does not exist.
  --> /home/bernhard/Projects/substrate/primitives/wasm-interface/src/lib.rs:73:56
   |
73 | #[derive(PartialEq, Debug, Clone, Copy, codec::Encode, codec::Decode)]
   |                                                        ^^^^^^^^^^^^^
   |
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)
# <snip>

there is a mix of stable native/host rust with nightly wasm .

@luser I could not find where the tree deps are included (are they) so I can iron this hopefully last pitfall out which I step into my daily usage :)

drahnr commented 4 years ago

Output of RUST_LOG=sccache=trace SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 sccache | rg -i -C 4 '(CWD|wasm|linux)'

TRACE 2020-03-05T09:04:45Z: sccache::compiler::rust: get_compiler_outputs: "libsp_wasm_interface-10e57696cdb0b389.rlib\n"
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: generate_hash_key took 0.366 s
TRACE 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Hash key: 00ac1b7796ee2e381768a71a4aa5ae7b076bcdab8cfd0dbbcec694390ef0f6cb7bacb5e326c336e1127cc580ac96ae9964b77f806a74c0b9689e650e469a0819
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Cache miss in 0.001 s
TRACE 2020-03-05T09:04:45Z: sccache::compiler::rust: [sp_wasm_interface]: compile
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Attempting distributed compilation
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Creating distributed compile request
TRACE 2020-03-05T09:04:45Z: sccache::compiler::rust: Dist inputs: inputs=["/home/bernhard/Projects/substrate/primitives/wasm-interface/src/lib.rs", "/home/bernhard/Projects/substrate/target/debug/wbuild/target/release/deps/libimpl_trait_for_tuples-73c1f651d00a57ac.so", "/home/bernhard/Projects/substrate/target/debug/wbuild/target/wasm32-unknown-unknown/release/deps/libparity_scale_codec-ae6fd645b0ad7e6a.rmeta", "/home/bernhard/Projects/substrate/target/debug/wbuild/target/wasm32-unknown-unknown/release/deps/libsp_std-fe408af98f20a1d8.rmeta"] crate_link_paths=["/home/bernhard/Projects/substrate/target/debug/wbuild/target/wasm32-unknown-unknown/release/deps", "/home/bernhard/Projects/substrate/target/debug/wbuild/target/release/deps"]
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Identifying dist toolchain for "/home/bernhard/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc"
DEBUG 2020-03-05T09:04:45Z: sccache::dist::cache::client: Using cached toolchain 1b938ab196be9665a3914d019a8e73565e621de792cbcd679de597d2e9d6ef455178f5a201b8c29703b88aab9ef43ebfa716b0616eed748eba7a503fb913ec45 -> 38c1b3a08da2bf48b2c1774dfe4aeb5db3131ddd6e6859de12bd774facdb695517a01270947450ab742b0622b5f1db98a46a1ebcfa35f7e3366d30c6a2c92919
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Requesting allocation
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::compiler: [sp_wasm_interface]: Running job
DEBUG 2020-03-05T09:04:45Z: sccache::compiler::rust: Packaging compile inputs for compile
TRACE 2020-03-05T09:04:45Z: sccache::compiler::rust: Identified dependency crate names: {"core", "byte_slice_cast", "rustc_std_workspace_core", "compiler_builtins", "alloc", "arrayvec", "parity_scale_codec_derive"}
TRACE 2020-03-05T09:04:45Z: sccache::dist::http::client: Compressed inputs from 24642048 -> 8403629
 INFO 2020-03-05T09:04:46Z: sccache::compiler::compiler: fetched [("/home/bernhard/Projects/substrate-/target/debug/wbuild/target/wasm32-unknown-unknown/release/deps/sp_wasm_interface-10e57696cdb0b389.d", "Size: 773->180")]
TRACE 2020-03-05T09:04:46Z: sccache::compiler::rust: Pondering on rewriting dep file Some("/home/bernhard/Projects/substrate/target/debug/wbuild/target/wasm32-unknown-unknown/release/deps/sp_wasm_interface-10e57696cdb0b389.d")
TRACE 2020-03-05T09:04:46Z: sccache::compiler::rust: Comparing with /home/bernhard/Projects/substrate/target/debug/wbuild/target/wasm32-unknown-unknown/release/deps/sp_wasm_interface-10e57696cdb0b389.d
 INFO 2020-03-05T09:04:46Z: sccache::compiler::rust: Replacing using the transformer PathTransformer
DEBUG 2020-03-05T09:04:46Z: sccache::compiler::compiler: [sp_wasm_interface]: Compiled but failed, not storing in cache
TRACE 2020-03-05T09:04:46Z: sccache::server: CompileFinished retcode: signal: 1
qTRACE 2020-03-05T09:06:45Z: sccache::server: incoming connection
TRACE 2020-03-05T09:06:45Z: sccache::server: handle_client
DEBUG 2020-03-05T09:06:45Z: sccache::server: handle_client: compile
TRACE 2020-03-05T09:06:45Z: sccache::server: compiler_info Adjusting current working directory for rustup rustc check "/home/bernhard/Projects/substrate" -> "/home/bernhard/Projects/substrate"
TRACE 2020-03-05T09:06:45Z: sccache::compiler::rust: rustup which rustc produced: "/home/bernhard/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc"
TRACE 2020-03-05T09:06:45Z: sccache::server: compiler_info cache hit
DEBUG 2020-03-05T09:06:45Z: sccache::server: check_compiler: Supported compiler
DEBUG 2020-03-05T09:06:45Z: sccache::server: parse_arguments: CannotCache(-): ["-", "--crate-name", "___", "--print=file-names", "--target", "x86_64-unknown-linux-gnu", "--crate-type", "bin", "--crate-type", "rlib", "--crate-type", "dylib", "--crate-type", "cdylib", "--crate-type", "staticlib", "--crate-type", "proc-macro", "--print=sysroot", "--print=cfg"]
luser commented 4 years ago

What does that codec derive do? Is the proc macro looking for the Cargo.toml file? For a distributed compile sccache does not send the Cargo.toml to the remote builder AFAIK. The code that packages inputs for distributed Rust compilation is here: https://github.com/mozilla/sccache/blob/41a584c081a37ffa08d86743b63c3ad300656408/src/compiler/rust.rs#L1493-L1683

drahnr commented 4 years ago

I think the root cause here is different.

A build.rs is triggering a cargo build from within that to create a wasm binary, to include in the final build via include!() macro.

Hence IIUC the easiest way would be to be able to prevent build.rs (or whatever is referenced in Cargo.toml) being excluded from dist compilation.

I would very happily implement that if that would be a desired change :)

drahnr commented 4 years ago

Possibly related to #675

luser commented 4 years ago

A build.rs is triggering a cargo build from within that to create a wasm binary, to include in the final build via include!() macro.

Oh, that's exciting. I would think that most build.rs compiles would not be distributed since they're usually producing a binary directly and sccache doesn't cache those compiles. Is the problem that it's distributing the compile driven by the build.rs and that is failing?

drahnr commented 4 years ago

I have yet to debug through this, the assumption currently is as you say: the build.rs is compiled via sccache-dist, the Cargo.toml it tries launch cargo for does not exist, hence it fails. The environment is passed (as in not modified, besides the target dir IIRC) to the subprocess. So that cargo instance launched from the build.rs is also aware of the env var RUSTC_WRAPPER, but it never gets that far. Hence my idea to exclude the build.rs by default, since it can contain all kind of mockery depending on the project setup and specifics (read git sha, inject things based on the current machine into the binary - while I do not recommend any of these, they are pretty common unfortunately).

luser commented 4 years ago

OK, I looked into this a little more. The codec derive is using this proc-macro-crate crate: https://github.com/paritytech/parity-scale-codec/blob/5f668d00b20fd0ad61e891b6483b3eefcde5e232/derive/Cargo.toml#L16 https://github.com/paritytech/parity-scale-codec/blob/5f668d00b20fd0ad61e891b6483b3eefcde5e232/derive/src/lib.rs#L42

That crate does in fact attempt to read Cargo.toml: https://github.com/bkchr/proc-macro-crate/blob/721d5a9dbdedbfbb6d160a3e4a482226d8edb178/src/lib.rs#L82

Since sccache does not send Cargo.toml to the build server this fails.

drahnr commented 4 years ago

So the issue that causes the failure is a different than the one I spotted: https://github.com/paritytech/substrate/blob/master/bin/node/runtime/build.rs https://github.com/paritytech/substrate/blob/master/utils/wasm-builder/src/prerequisites.rs#L63-L65 https://github.com/paritytech/substrate/blob/master/utils/wasm-builder/src/lib.rs#L188-L194 which would fail as I described before, but that is worth a separate ticket I guess

DemiMarie commented 4 years ago

I would prefer extra build requirements to be explicitly added.

luser commented 4 years ago

This likely needs upstream rustc changes to make it possible for sccache to know what's happening. There's https://github.com/rust-lang/rust/issues/73921, which proposes adding an API that proc macros could use to indicate that they use external files. If that was added and proc macros like the one mentioned here used it then rustc could list these files in --output=dep-info. sccache already parses that output to use as inputs to the hash key so it could also use it to gather files for distributed compilation.

Side note: this made me realize that this is also currently a caching hazard in that sccache doesn't know to include files like this in the hash, so it could produce incorrect results from the cache.