sccache v0.4.0 failed due to invalid `-E` options

[NobodyXu]

This bug is discovered in https://github.com/rust-secure-code/cargo-auditable/issues/87

It appears that sccache calls the Rust workspace wrapper with the arguments intended for the C compiler instead of the Rust compiler. It invokes /home/runner/.cargo/bin/cargo-auditable-v0.6.1 -E /tmp/sccachemNgxBK/testfile.c, which is not a valid call to rustc. The Rust compiler does not accept the -E option (gcc does), and it's being passed a C file, not a Rust file.

There's nothing I can do to fix this in cargo auditable. You'll have to file a bug against sccache. I'll research workarounds so you could get the caching going without waiting on a fix.

[Shnatsel]

A bit of context: cargo-auditable sets itself as the RUSTC_WORKSPACE_WRAPPER, proper support for which has been added to sccache in https://github.com/mozilla/sccache/pull/1280

There's detailed technical info in the linked issue, starting here.

[NobodyXu]

Tried sccache v0.4.0, still same error.

[NobodyXu]

P.S. The bug was found in https://github.com/cargo-bins/cargo-quickinstall/pull/206

[NobodyXu]

The same error now also occurs in cargo-binstall https://github.com/cargo-bins/cargo-binstall/pull/934 even without cargo-auditable:

cargo test  --target x86_64-unknown-linux-gnu --no-default-features --features rustls,fancy-with-backtrace,zstd-thin,log_release_max_level_debug,pkg-config
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `sccache rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 2)
  --- stderr
  sccache: error: failed to execute compile
  sccache: caused by: Failed to send data to or receive data from server
  sccache: caused by: Failed to read response header
  sccache: caused by: failed to fill whole buffer
error: Recipe `unit-tests` failed on line 209 with exit code 101
Error: Process completed with exit code 101.

And also from running sccache --show-stats:

/home/runner/.cargo/bin/sccache --show-stats
sccache: error: failed to execute compile
sccache: caused by: Compiler not supported: "error: Found argument \'-E\' which wasn\'t expected, or isn\'t valid in this context\n\n  If you tried to supply \'-E\' as a value rather than a flag, use \'-- -E\'\n\nUsage: sccache [OPTIONS] <--dist-auth|--dist-status|--show-stats|--start-server|--stop-server|--zero-stats|--package-toolchain <EXE> <OUT>|CMD>\n\nFor more information try \'--help\'\n"
/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1702
                error = new Error(`The process '${this.toolPath}' failed with exit code ${this.processExitCode}`);
                        ^

Error: The process '/home/runner/.cargo/bin/sccache' failed with exit code 2
    at ExecState._setResult (/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1702:25)
    at ExecState.CheckComplete (/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1685:18)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1579:27)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

From CI " test (x86_64-unknown-linux-gnu, ubuntu-latest)"

[NobodyXu]

This error only occurs on Linux and does not occur on MacOS or Windows.

cc @Xuanwo @sylvestre

[NobodyXu]

Also, cargo-binstall seems to work fine when using sccache v0.4.0-pre.10 for caching, so this might also be a regression.

[Xuanwo]

Yes, I think it's a sccache bug. sccache --version is broken too:

:) sccache --version
sccache: error: failed to execute compile
sccache: caused by: Compiler not supported: "error: Found argument \'-E\' which wasn\'t expected, or isn\'t valid in this context\n\n  If you tried to supply \'-E\' as a value rather than a flag, use \'-- -E\'\n\nUsage: sccache [OPTIONS] <--dist-auth|--dist-status|--show-stats|--start-server|--stop-server|--zero-stats|--package-toolchain <EXE> <OUT>|CMD>\n\nFor more information try \'--help\'\n"

But I am unable to reproduce it by

• building from the source of main.
• buidling from the source of tag v0.4
• download the binary from github releases.

But I can reproduce it via install via cargo binstall sccache.

[NobodyXu]

But I am unable to reproduce it by building from the source.

@Shnatsel was able to reproduce this using the command from https://github.com/rust-secure-code/cargo-auditable/issues/87#issuecomment-1474749551

For cargo-binstall CI failure, you can try reproducing using:

RUSTC_WRAPPER=sccache CARGO_INCREMENTAL=0 cargo test  --target x86_64-unknown-linux-gnu --no-default-features --features rustls,fancy-with-backtrace,zstd-thin,log_release_max_level_debug,pkg-config

which is almost identical to the command we run on the CI except the use of GHA for caching.

[Xuanwo]

But I am unable to reproduce it by

• building from the source of main.
• buidling from the source of tag v0.4
• download the binary from github releases.

But I can reproduce it via install via cargo binstall sccache.

The minimal repro looks like the following:

> rm ~/.cargo/bin/sccache
> cargo install sccache --force
> sccache --version <------- works
sccache 0.4.0

> rm ~/.cargo/bin/sccache
> cargo binstall sccache --force
> sccache --version <------- broken
sccache: error: failed to execute compile
sccache: caused by: Compiler not supported: "error: Found argument \'-E\' which wasn\'t expected, or isn\'t valid in this context\n\n  If you tried to supply \'-E\' as a value rather than a flag, use \'-- -E\'\n\nUsage: sccache [OPTIONS] <--dist-auth|--dist-status|--show-stats|--start-server|--stop-server|--zero-stats|--package-toolchain <EXE> <OUT>|CMD>\n\nFor more information try \'--help\'\n"

[Xuanwo]

Hi, @NobodyXu, is anything special to install sccache from cargo-binstall?

:( paru -Q cargo-binstall
cargo-binstall 0.21.3-1

[NobodyXu]

@Xuanwo No, it simply downloads and unpacks pre-built sccache from mozilla/sccache github release.

[Xuanwo]

Hi, @NobodyXu, is anything special to install sccache from cargo-binstall?

Oh, I got it. cargo-binstall will create a symlink for sccache:

lrwxrwxrwx    14 xuanwo 22 Mar 14:11  sccache -> sccache-v0.4.0
.rwxr-xr-x   14M xuanwo 21 Mar 20:50  sccache-v0.4.0

Sadly, it doesn't work on sccache. sccache requires to be called as sccache.

[Xuanwo]

After remove the symlink and rename sccache-v0.4.0 to sccache, it works again.

There is a short doc for this: https://github.com/mozilla/sccache#symbolic-links

[NobodyXu]

Thanks!

Also, the same error occurs on cargo-quickinstall where we use mozilla-actions/sccache-action@v0.0.2

https://github.com/rust-secure-code/cargo-auditable/issues/87#issuecomment-1474611491.

[Xuanwo]

Also, the same error occurs on cargo-quickinstall where we use mozilla-actions/sccache-action@v0.0.2

I think they are facing the same issues. Perhaps sccache can provide a clearer error message for this situation.

[NobodyXu]

@Xuanwo But mozilla-actions/sccache-action@v0.0.2 does not install a symlink, though cargo-auditable is installed with a symlink but TBF rustc itself is also a symlink.

[Xuanwo]

@Xuanwo But mozilla-actions/sccache-action@v0.0.2 does not install a symlink, though cargo-auditable is installed with a symlink but TBF rustc itself is also a symlink.

Yep, you are correct. https://github.com/cargo-bins/cargo-quickinstall/pull/206 is another issue. I'll take a look now.

[Xuanwo]

Yep, you are correct. cargo-bins/cargo-quickinstall#206 is another issue. I'll take a look now.

I'm not sure about this:

 INFO This will install the following binaries:
 INFO   - cargo-auditable (cargo-auditable -> /home/runner/.cargo/bin/cargo-auditable-v0.6.1)
 INFO And create (or update) the following symlinks:
 INFO   - cargo-auditable (/home/runner/.cargo/bin/cargo-auditable -> cargo-auditable-v0.6.1)

Can you try install cargo-auditable directly? rustc is Ok to be a symlink.

[Xuanwo]

The code related to symlinks was written six years ago. I need some time to revisit it and see if we can improve it.

[Xuanwo]

Oh, I got it. cargo-binstall will create a symlink for sccache:

lrwxrwxrwx    14 xuanwo 22 Mar 14:11  sccache -> sccache-v0.4.0
.rwxr-xr-x   14M xuanwo 21 Mar 20:50  sccache-v0.4.0

Sadly, it doesn't work on sccache. sccache requires to be called as sccache.

Hi, @NobodyXu, is it a good idea for you to change symlink into hardlink on linux platform? This may be the quickest solution so far. This change can help users who use cargo-binstall to install sccache. Sorry for asking those changes because of the sccache behavior.

[NobodyXu]

Hi, @NobodyXu, is it a good idea for you to change symlink into hardlink on linux platform? This may be the quickest solution so far. This change can help users who use cargo-binstall to install sccache.

We actually have a tracking issue to enable --no-symlinks by default https://github.com/cargo-bins/cargo-binstall/issues/731

[NobodyXu]

We actually have a tracking issue to enable --no-symlinks by default https://github.com/cargo-bins/cargo-binstall/issues/731

Opened https://github.com/cargo-bins/cargo-binstall/pull/936 for this

[Xuanwo]

Opened cargo-bins/cargo-binstall#936 for this

Thank you very much for considering my advice!

[NobodyXu]

@Xuanwo I tried install cargo-auditable with --no-symlinks but it still failed with the same error in CI:

error: failed to compile `cargo-deny v0.13.7`, intermediate artifacts can be found at `/tmp/cargo-install4jaa7D`

Caused by:
  failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `sccache /home/runner/.cargo/bin/cargo-auditable rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 2)
  --- stderr
  sccache: error: failed to execute compile
  sccache: caused by: Compiler not supported: "thread \'main\' panicked at \'Failed to invoke rustc! Make sure it\'s in your $PATH: Os { code: 2, kind: NotFound, message: \"No such file or directory\" }\', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo-auditable-0.6.1/src/rustc_wrapper.rs:90:10\nstack backtrace:\n   0: rust_begin_unwind\n             at ./rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:575:5\n   1: core::panicking::panic_fmt\n             at ./rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panicking.rs:64:14\n   2: core::result::unwrap_failed\n             at ./rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/result.rs:1791:5\n   3: cargo_auditable::main\nnote: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.\n"

NOTE that it fails on Linux, MacOS and Windows all with the same error.

[NobodyXu]

Thank you very much for considering my advice!

No worries, it's actually long overdue.

[Xuanwo]

NOTE that it fails on Linux, MacOS and Windows all with the same error.

Yes, it's another issue and I am still investigating it.

[drahnr]

We actually have a tracking issue to enable --no-symlinks by default cargo-bins/cargo-binstall#731

Opened cargo-bins/cargo-binstall#936 for this

The short term solution is to mention in the README.md install with binstall is only supported with --no-symlinks. And then investigate further why the symlinks are such a headache.

[NobodyXu]

The short term solution is to mention in the README.md install with binstall is only supported with --no-symlinks. And then investigate further why the symlinks are such a headache.

I plan to publish a new release v0.22.0 once this PR and few others trivial PRs get merged, likely in 1 day or two.

[NobodyXu]

@drahnr cargo-binstall v0.22.0 has released.

[NobodyXu]

NOTE that it fails on Linux, MacOS and Windows all with the same error.

Yes, it's another issue and I am still investigating it.

@Xuanwo Friendly ping on this issue. Is there any progress on this one?

[Xuanwo]

Is there any progress on this one?

Sorry for the delay. I haven't been working on this issue in recent days, so there has been no progress thus far.

[NobodyXu]

@Xuanwo I tried sccache v0.5.3 and this bug is still reproducible.

[Xuanwo]

@Xuanwo I tried sccache v0.5.3 and this bug is still reproducible.

Sorry, but I'm not working on this issue these days.

[NobodyXu]

@Xuanwo It's ok, I just want to try if latest sccache version has fixed this bug or not.

[Xuanwo]

@Xuanwo It's ok, I just want to try if latest sccache version has fixed this bug or not.

We will ping this issue if it got fixed.

[NobodyXu]

Thank you @Xuanwo