warning: failed to connect to jobserver from environment variable

[VorpalBlade]

Using RUSTFLAGS="-Zcodegen-backend=cranelift" cargo +nightly build I get a lot of the following warnings building my project:

warning: failed to connect to jobserver from environment variable `CARGO_MAKEFLAGS="-j --jobserver-fds=3,4 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured

This doesn't appear when building with the standard native LLVM toolchain on either nightly or stable.

Version info:

$ rustup --version
rustup 1.26.0 (2023-11-14)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.74.1 (a28077b28 2023-12-04)`

$ rustc +nightly --version --verbose
rustc 1.76.0-nightly (de686cbc6 2023-12-14)
binary: rustc
commit-hash: de686cbc65478db53e3d51c52497685e852cc092
commit-date: 2023-12-14
host: x86_64-unknown-linux-gnu
release: 1.76.0-nightly
LLVM version: 17.0.5

Project for reproduction (haven't tried it on other things, don't know how widespread it is): https://github.com/VorpalBlade/chezmoi_modify_manager (tested on commit ec6cc2477dd5dca4fff5d2b972aa18ced86df9b8).

[VorpalBlade]

Further note, it seems it is always associated with the same crates. Not sure what the pattern is. Here are some examples. Two are proc-macros, but some of the other ones aren't:

   Compiling serde_derive v1.0.193
warning: failed to connect to jobserver from environment variable `CARGO_MAKEFLAGS="-j --jobserver-fds=3,4 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured
[...]
   Compiling enumflags2_derive v0.7.8
warning: failed to connect to jobserver from environment variable `CARGO_MAKEFLAGS="-j --jobserver-fds=3,4 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured
[...]
Compiling rand v0.8.5
warning: failed to connect to jobserver from environment variable `CARGO_MAKEFLAGS="-j --jobserver-fds=3,4 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured

warning: failed to connect to jobserver from environment variable `CARGO_MAKEFLAGS="-j --jobserver-fds=3,4 --jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9)
  |
  = note: the build environment is likely misconfigured

[bjorn3]

I have no clue what causes this. The only place where this error is triggered should be when rustc itself is creating the compiler session, which is before cg_clif runs.

[Nemo157]

I think this may be related to inline global asm, doing some strace'ng I see this just before one of the warnings:

[pid 3627197] execve("/nix/store/p7ca2l0faqcqj1bkv447azi12pcfqfb9-rust-default-1.77.0-nightly-2023-12-25/bin/rustc", ["/nix/store/p7ca2l0faqcqj1bkv447a"..., "--target", "x86_64-unknown-linux-gnu", "--crate-type", "staticlib", "--emit", "obj", "-o", "/run/user/1000/cargo-home/target"..., "-", "-Abad_asm_style", "-Zcodegen-backend=llvm"], 0x7fffffff7308 /* 223 vars */) = 0

https://github.com/rust-lang/rustc_codegen_cranelift/blob/39f0dac77d12dfa4d84eb832ca9143b37453a2bc/src/global_asm.rs#L156

[VorpalBlade]

https://github.com/rust-lang/rustc_codegen_cranelift/blob/39f0dac77d12dfa4d84eb832ca9143b37453a2bc/src/global_asm.rs#L156

That seems like a massive hack. Am I reading this correctly that it simply calls out to rustc llvm backend for global assembly? Won't these extra processes cause a significant slowdown in build times (which I thought was the main point of this backend compared to llvm)? Probably will be even worse on Windows (iirc spawning processes there is much slower than on Linux, but it has been a while since I used Windows).

EDIT: Also that raw string hack is brittle. It should ideally search for the longest occurance of consecutive # in the assembly string and use at least one more than that. Might come up in assembly declared data variables, which iirc allows to declare string data. Though probably not in normal assembly? Caveat: I haven't written assembler for nearly a decade now, so my memory is a bit fuzzy on this, plus I don't know the LLVM variant of assembler at all really.

[bjorn3]

Am I reading this correctly that it simply calls out to rustc llvm backend for global assembly?

Yes

Won't these extra processes cause a significant slowdown in build times

Yes, but only if you actually use inline asm. And also I'm using #![no_core] to improve performance by skipping loading of the standard library crate metadata.

Probably will be even worse on Windows

Windows compatibility is the whole reason I'm doing this. On Linux and macOS I could call into the gnu assembler as it is installed by default on most systems together with the rest of the C compiler toolchain, but Windows doesn't ship with it and neither does rustc. MSVC does have an assembler, but it's assembly syntax is not compatible with the gnu assembler and thus not usable for implementing rust inline asm.

[VorpalBlade]

Windows compatibility is the whole reason I'm doing this. [...]

Ah that makes sense. Sigh.

Yes, but only if you actually use inline asm.

To be fair, I don't use inline or global assembly. But apparently several of my dependencies do? I don't think I'm using anything really unusual, so it is probably some foundational crate that is doing this in the name of performance, though I thought most of them used intrinsics instead for SIMD.

So that means many full builds will be affected by this probably?

[bjorn3]

Removing the CARGO_MAKEFLAGS env var did the trick. Thanks @Nemo157 for pointing me in the right direction!

[bjorn3]

so it is probably some foundational crate that is doing this in the name of performance, though I thought most of them used intrinsics instead for SIMD.

A couple of intrinsics are implemented in cg_clif using inline asm because they are way to complex to emulate using handrolled clif ir and cranelift doesn't have instructions that directly lower to the respective cpu instructions.

[dimitri-lenkov]

Hi @bjorn3,

Removing the CARGO_MAKEFLAGS env var did the trick. Thanks @Nemo157 for pointing me in the right direction!

Thank you for finding a solution! I'm running into the same warnings.

I tried calling unset CARGO_MAKEFLAGS before running cargo run but I'm still getting the warnings.

How did you remove this env var?

[bjorn3]

Cargo sets this env var for every rustc it runs. I changed cg_clif itself to remove it before spawning another rustc instance. It hasn't yet been synced back to the rust repo yet, so rustc nightlies don't have the fix yet.

[bjorn3]

The fix should be on nightly now.

[dimitri-lenkov]

The fix should be on nightly now.

:pray: thank you :bow: