sudo cargo install recompile everything after cargo build --release

12101111 commented 2 years ago

Problem

Users may want to install a rust binary crate to /usr/local/bin, not to $HOME/.cargo/bin, and this can be archived by --root flag of cargo install.

And normal users don't have the permission to write /usr/local/bin directly, so they have to run sudo cargo install --root /usr/local instead.

The bug is here: when cargo is running as another user, it will recompile everything due to the environment change.

Steps

> #unset anything tracked by cargo
> $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo new hello
      Created binary (application) `hello` package
> $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo build --release
   Compiling hello v0.1.0 (/tmp/hello)
    Finished release [optimized] target(s) in 0.81s
> $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo install --path ./ --root /tmp/usr    
 Installing hello v0.1.0 (/tmp/hello)
    Finished release [optimized] target(s) in 0.01s
   Replacing /tmp/usr/bin/hello
    Replaced package `hello v0.1.0 (/tmp/hello)` with `hello v0.1.0 (/tmp/hello)` (executable `hello`)
warning: be sure to add `/tmp/usr/bin` to your PATH to be able to run the installed binaries
> $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo build --release
    Finished release [optimized] target(s) in 0.01s
> sudo $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo install --path ./ --root /tmp/usr
  Installing hello v0.1.0 (/tmp/hello)
   Compiling hello v0.1.0 (/tmp/hello)
    Finished release [optimized] target(s) in 5.27s
   Replacing /tmp/usr/bin/hello
    Replaced package `hello v0.1.0 (/tmp/hello)` with `hello v0.1.0 (/tmp/hello)` (executable `hello`)
warning: be sure to add `/tmp/usr/bin` to your PATH to be able to run the installed binaries

( Don't use sudo cargo directly when there is another system wide rust installation )

Possible Solution(s)

cargo should detect this situation.

A fallback solution is adding a new flags like "install-only" that don't recompile no matter the environment is changed or not.

Notes

cmake & ninja won't recompile everything when run as root, so you can use

mkdir build && cd build && cmake .. -GNinja
ninja
sudo ninja install

to install any cmake based project.

I first notice this issue when I compile some rust packages using Portage (gentoo's package manager).

>>> Source unpacked in /tmp/portage/sys-apps/bat-0.20.0/work
>>> Preparing source in /tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0 ...
>>> Source prepared.
>>> Configuring source in /tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0 ...
>>> Source configured.
>>> Compiling source in /tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0 ...
 * cargo build --release
   Compiling libc v0.2.112
   Compiling proc-macro2 v1.0.36
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.85
   Compiling pkg-config v0.3.24
   Compiling autocfg v1.0.1
   Compiling serde_derive v1.0.136
   Compiling cfg-if v1.0.0
   Compiling memchr v2.4.1
   Compiling serde v1.0.136
   Compiling encoding_index_tests v0.1.4
   Compiling regex-syntax v0.6.25
   Compiling proc-macro-hack v0.5.19
   Compiling lazy_static v1.4.0
   Compiling log v0.4.14
   Compiling tinyvec_macros v0.1.0
   Compiling matches v0.1.9
   Compiling unicode-width v0.1.9
   Compiling bitflags v1.3.2
   Compiling crc32fast v1.3.0
   Compiling hashbrown v0.11.2
   Compiling ryu v1.0.9
   Compiling itoa v0.4.8
   Compiling regex-automata v0.1.10
   Compiling vec_map v0.8.2
   Compiling adler v1.0.2
   Compiling fnv v1.0.7
   Compiling ansi_term v0.12.1
   Compiling same-file v1.0.6
   Compiling strsim v0.8.0
   Compiling linked-hash-map v0.5.4
   Compiling percent-encoding v2.1.0
   Compiling safemem v0.3.3
   Compiling serde_json v1.0.74
   Compiling unicode-bidi v0.3.7
   Compiling base64 v0.13.0
   Compiling semver v1.0.4
   Compiling itoa v1.0.1
   Compiling bytemuck v1.7.3
   Compiling bugreport v0.4.1
   Compiling xml-rs v0.8.4
   Compiling once_cell v1.9.0
   Compiling std_prelude v0.2.12
   Compiling lazycell v1.3.0
   Compiling termcolor v1.1.2
   Compiling shell-escape v0.1.5
   Compiling shell-words v1.0.0
   Compiling bytesize v1.1.0
   Compiling wild v2.0.4
   Compiling encoding-index-singlebyte v1.20141219.5
   Compiling encoding-index-simpchinese v1.20141219.5
   Compiling encoding-index-korean v1.20141219.5
   Compiling encoding-index-japanese v1.20141219.5
   Compiling encoding-index-tradchinese v1.20141219.5
   Compiling tinyvec v1.5.1
   Compiling indexmap v1.7.0
   Compiling miniz_oxide v0.4.4
   Compiling walkdir v2.3.2
   Compiling form_urlencoded v1.0.1
   Compiling yaml-rust v0.4.5
   Compiling line-wrap v0.1.1
   Compiling rgb v0.8.31
   Compiling path_abs v0.5.1
   Compiling encoding v0.2.33
   Compiling unicode-normalization v0.1.19
   Compiling ansi_colours v1.1.1
   Compiling aho-corasick v0.7.18
   Compiling bstr v0.2.17
   Compiling content_inspector v0.2.4
   Compiling quote v1.0.14
   Compiling jobserver v0.1.24
   Compiling term_size v0.3.2
   Compiling atty v0.2.14
   Compiling time v0.3.5
   Compiling dirs-sys-next v0.1.2
   Compiling terminal_size v0.1.17
   Compiling textwrap v0.11.0
   Compiling idna v0.2.3
   Compiling cc v1.0.72
   Compiling dirs-next v2.0.0
   Compiling regex v1.5.4
   Compiling clap v2.34.0
   Compiling url v2.2.2
   Compiling globset v0.4.8
   Compiling console v0.15.0
   Compiling flate2 v1.0.22
   Compiling grep-cli v0.1.6
   Compiling libz-sys v1.1.3
   Compiling onig_sys v69.7.1
   Compiling sys-info v0.9.1
   Compiling libgit2-sys v0.12.26+1.3.0
   Compiling onig v6.3.1
   Compiling bat v0.20.0 (/tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0)
   Compiling git2 v0.13.25
   Compiling git-version-macro v0.3.5
   Compiling thiserror-impl v1.0.30
   Compiling thiserror v1.0.30
   Compiling git-version v0.3.5
   Compiling bincode v1.3.3
   Compiling plist v1.3.1
   Compiling clircle v0.3.0
   Compiling serde_yaml v0.8.23
   Compiling syntect v4.6.0
    Finished release [optimized] target(s) in 3m 06s
>>> Source compiled.
>>> Test phase [not enabled]: sys-apps/bat-0.20.0

>>> Install sys-apps/bat-0.20.0 into /tmp/portage/sys-apps/bat-0.20.0/image
 * cargo install --path ./ --root /tmp/portage/sys-apps/bat-0.20.0/image/usr
  Installing bat v0.20.0 (/tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0)
   Compiling libc v0.2.112
   Compiling syn v1.0.85
   Compiling atty v0.2.14
   Compiling term_size v0.3.2
   Compiling time v0.3.5
   Compiling terminal_size v0.1.17
   Compiling flate2 v1.0.22
   Compiling dirs-sys-next v0.1.2
   Compiling jobserver v0.1.24
   Compiling grep-cli v0.1.6
   Compiling textwrap v0.11.0
   Compiling console v0.15.0
   Compiling dirs-next v2.0.0
   Compiling cc v1.0.72
   Compiling clap v2.34.0
   Compiling libz-sys v1.1.3
   Compiling onig_sys v69.7.1
   Compiling sys-info v0.9.1
   Compiling libgit2-sys v0.12.26+1.3.0
   Compiling bugreport v0.4.1
   Compiling serde_derive v1.0.136
   Compiling git-version-macro v0.3.5
   Compiling thiserror-impl v1.0.30
   Compiling onig v6.3.1
   Compiling git2 v0.13.25
   Compiling bat v0.20.0 (/tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0)
   Compiling git-version v0.3.5
   Compiling thiserror v1.0.30
   Compiling serde v1.0.136
   Compiling bincode v1.3.3
   Compiling plist v1.3.1
   Compiling serde_json v1.0.74
   Compiling serde_yaml v0.8.23
   Compiling clircle v0.3.0
   Compiling syntect v4.6.0
    Finished release [optimized] target(s) in 2m 08s
  Installing /tmp/portage/sys-apps/bat-0.20.0/image/usr/bin/bat
   Installed package `bat v0.20.0 (/tmp/portage/sys-apps/bat-0.20.0/work/bat-0.20.0)` (executable `bat`)
warning: be sure to add `/tmp/portage/sys-apps/bat-0.20.0/image/usr/bin` to your PATH to be able to run the installed binaries
>>> Completed installing sys-apps/bat-0.20.0 into /tmp/portage/sys-apps/bat-0.20.0/image

 * Final size of build directory: 521044 KiB (508.8 MiB)
 * Final size of installed tree:    4240 KiB (  4.1 MiB)

Note that only a few crates is recompiled.

Version

> $HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo version --verbose
cargo 1.61.0-nightly (109bfbd05 2022-03-17)
release: 1.61.0-nightly
commit-hash: 109bfbd055325ef87a6e7f63d67da7e838f8300b
commit-date: 2022-03-17
host: x86_64-unknown-linux-musl
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux 2.8 [64-bit]

> /usr/bin/cargo version --verbose
cargo 1.59.0-nightly
release: 1.59.0-nightly
host: x86_64-unknown-linux-musl
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.82.0 (sys:0.4.51+curl-7.80.0 system ssl:OpenSSL/1.1.1n)
os: Linux 2.8 [64-bit]

weihanglo commented 2 years ago

The fingerprint that determines a rebuild is affected by several things. I guess at least the version info of invoked rustc changed when switching to a super user, due to PATH changed. That behaviour is expected because when across different version of rustc, we need to recompile.

The Cargo Book contains a whole paragraph about knowing the reason why a rebuild occurs. It's not comprehensive but usable for now. You could test it to help verify my assumption.

If I do want to avoid recompilation when sudo cargo, I might set CARGO_HOME=/path/to/my/cargo/bin/dir and then invoke cargo in order to control which rustc to be called. (I haven't tried it, though)

12101111 commented 2 years ago

I find that the exa case is not a fingerprint bug, but (maybe) a old bug of feature resolver v1.

I hack cargo to print build plan when run cargo install subcommand, and compare this output with build plan from cargo build -Z unstable-options --build-plan

The different is, in build plan of cargo install, --cfg feature="extra_traits" is enabled for libc and --cfg feature="full" is enabled for syn, both of them is pulled by a dev-dependency ( nix need extra_traits feature of libc, serial_test_derive need full feature of syn)

So the rebuild is caused by behavior mismatch between cargo install and cargo build in feature resolver v1.

weihanglo commented 2 years ago

Hmm… interesting. Haven't tried out your new discovery, but I can reproduce the steps in the PR description and the example has no dependency, so these two might be different issues.

If you can set identical CARGO_HOME or PATH for the superuser account, the build should be fresh and won't trigger any re-build I guess (for the example in PR description).

ehuss commented 2 years ago

It looks like this is using cargo install without the --locked option. cargo install will by default ignore Cargo.lock, which is probably what you are seeing here.

12101111 commented 2 years ago

It's cause by this:

https://github.com/rust-lang/cargo/blob/403c6bd257e4501ec59d51aadb347d83ca0f2fb3/src/cargo/ops/cargo_install.rs#L755

And introduced in https://github.com/rust-lang/cargo/commit/d1ef031a6272e9eee55d97189c1f02bd24f1504a

YuanYuYuan commented 1 year ago

For someone who encountered the same problem, here's a quick solution.

sudo env "CARGO_HOME=$HOME/.cargo" cargo build         # or run or install

rust-lang / cargo