jo-so
commented
7 months ago BTW: This bug leaves you with a state where you can't do anything with rustup, no update, no compile of any crate (e.g. rustup itself) to debug.
Closed jo-so closed 7 months ago
jo-so
commented
7 months ago BTW: This bug leaves you with a state where you can't do anything with rustup, no update, no compile of any crate (e.g. rustup itself) to debug.
rami3l
commented
7 months ago @jo-so Thanks a lot for filing this issue!
I guess this is a duplicate of #3344, so I suggest we continue the discussion over there. Please feel free to correct me and reopen this issue if that's not the case.
jo-so
commented
7 months ago @rami3l I don't think so. #3344 discusses about symlinks for downloads and tmp. This bug report is about toolchains. And it worked since last week or so, while #3344 is open for a longer time.
rami3l
commented
7 months ago @jo-so Okay, I'll keep both issues open but those two issues do seem related. Unfortunately since this is caused by self-referential symlinks (created presumably in another rustup run other than the current one), your strace isn't very helpful.
I'll find some time to investigate into both issues.
jo-so
commented
7 months ago Here's a git-bisect run. A good version was 1.25.0, and the culprit is 2c9297d51a077d9a364e98ad8ebaee2301cd0c0d.
# bad: [1c612cf65439901200f89c0123dbdf9dcae5e539] fix(ci): fix file paths in CI-generated `*.sha256` files on *nix
# good: [976d72a1920a0266936320afc63b5e651888359f] (changelog): Fix typo
git bisect start 'origin/master' '1.25.0'
# skip: [44c7d94079d19dacdf63fd90cf16297df6293843] Merge pull request #3338 from hi-rustin/rustin-patch-clippy-tip
git bisect skip 44c7d94079d19dacdf63fd90cf16297df6293843
# good: [060b8f9e3f5a8a810b49108aa57cee2e784ee104] Merge pull request #3054 from MoSal/master
git bisect good 060b8f9e3f5a8a810b49108aa57cee2e784ee104
# bad: [d63b6e5f7608369934a4774611af9a83104d3a78] Address `#[warn(clippy::useless_conversion)]`
git bisect bad d63b6e5f7608369934a4774611af9a83104d3a78
# good: [de9a9f08c8b33f58541aa3df0ee80c5244bd5354] Merge pull request #3252 from hi-rustin/rustin-patch-actions-run-test
git bisect good de9a9f08c8b33f58541aa3df0ee80c5244bd5354
# good: [41b631c778d4806f2739e81fc7f4d7b765b8c8b9] Merge pull request #3333 from workingjubilee/use-mutex-const-new
git bisect good 41b631c778d4806f2739e81fc7f4d7b765b8c8b9
# bad: [8a704172114133095086c11b649afed6a64b3d95] CI support for loongarch64-unknown-linux-gnu
git bisect bad 8a704172114133095086c11b649afed6a64b3d95
# good: [395e49270394680dd03c74f69dcfedc6c3283ab5] Suggest right toolchain when running clippy
git bisect good 395e49270394680dd03c74f69dcfedc6c3283ab5
# bad: [2c9297d51a077d9a364e98ad8ebaee2301cd0c0d] Rework Toolchain model and drop relative file path overrides
git bisect bad 2c9297d51a077d9a364e98ad8ebaee2301cd0c0d
# good: [46812bf38a66c51558d4036c630d6640ead45492] Merge pull request #3335 from xfix/toml-0.7.3
git bisect good 46812bf38a66c51558d4036c630d6640ead45492
# good: [8662c33cc983a8fc0690ebb536cca91e027af094] Merge pull request #3343 from cuishuang/master
git bisect good 8662c33cc983a8fc0690ebb536cca91e027af094
# good: [4b29b79af7fe09cc7911528257241be5fbba25c6] Merge pull request #3287 from rbtcollins/tracing
git bisect good 4b29b79af7fe09cc7911528257241be5fbba25c6
# first bad commit: [2c9297d51a077d9a364e98ad8ebaee2301cd0c0d] Rework Toolchain model and drop relative file path overrides@jo-so Okay, I'll keep both issues open but those two issues do seem related. Unfortunately since this is caused by self-referential symlinks (created presumably in another
rustuprun other than the current one), yourstraceisn't very helpful.I'll find some time to investigate into both issues.
It's not about self-referential symlinks. As you can see in the strace the flag O_NOFOLLOW gets passed to the openat, which causes not even one symlink is crossed.
With a self compiled version you get
% RUSTUP_FORCE_ARG0=rustup RUSTUP_HOME=/home/joerg/kein_Backup/rustup-test strace -Zk -e t=openat target/debug/rustup-init update
11:59:57.099680 openat(AT_FDCWD</home/joerg/git/rustup>, "/home/joerg/kein_Backup/rustup-test/toolchains", O_RDONLY|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of symbolic links) <0.000032>
> /usr/lib/x86_64-linux-gnu/libc.so.6(__open64+0x51) [0xf7861]
> /home/joerg/git/rustup/target/debug/rustup-init(std::sys::pal::unix::fs::File::open_c+0xfc) [0x10d489c]
> /home/joerg/git/rustup/target/debug/rustup-init(std::fs::OpenOptions::_open+0x8b) [0x10c73fb]
> /home/joerg/git/rustup/target/debug/rustup-init(std::fs::OpenOptions::open+0x73) [0x4f4223]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::utils::raw::open_dir+0x64) [0x33d3c4]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::toolchain::toolchain::Toolchain::exists+0x296) [0x1dbe16]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::toolchain::toolchain::Toolchain::new+0xa8) [0x1db848]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::toolchain::distributable::DistributableToolchain::new+0x95) [0x25bfe5]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::config::Cfg::list_channels::{{closure}}+0x80) [0x2f8ba0]
> /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::map::map_try_fold::{{closure}}+0x69) [0x3637e9]
> /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::filter::filter_try_fold::{{closure}}+0x121) [0x1a2f21]
> /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::filter_map::filter_map_try_fold::{{closure}}+0x14c) [0x2e7bac]
> /home/joerg/git/rustup/target/debug/rustup-init(core::iter::traits::iterator::Iterator::try_fold+0x10f) [0x34e11f]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::filter_map::FilterMap<I,F> as core::iter::traits::iterator::Iterator>::try_fold+0x4f) [0x2e692f]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold+0x47) [0x1a1ac7]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold+0x40) [0x361230]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold+0x4f) [0x30611f]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next+0x24) [0x305e84]
> /home/joerg/git/rustup/target/debug/rustup-init(<alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter+0x53) [0x1f22b3]
> /home/joerg/git/rustup/target/debug/rustup-init(alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter+0x1e) [0x1fcc1e]
> /home/joerg/git/rustup/target/debug/rustup-init(<alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter+0x29) [0x2000d9]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}+0x22) [0x1b8302]
> /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::try_process+0x74) [0x306a44]
> /home/joerg/git/rustup/target/debug/rustup-init(<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter+0x27) [0x1b8287]
> /home/joerg/git/rustup/target/debug/rustup-init(core::iter::traits::iterator::Iterator::collect+0xe) [0x36283e]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::config::Cfg::list_channels+0x106) [0x19f216]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::config::Cfg::update_all_channels+0x3f) [0x19f2df]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::cli::common::update_all_channels+0x54) [0x18d5e4]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::cli::rustup_mode::update+0x43f) [0x2343ef]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::cli::rustup_mode::main+0xd62) [0x226412]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::run_rustup_inner+0x38b) [0x1813bb]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::run_rustup+0x1a2) [0x180ef2]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::maybe_trace_rustup+0xd) [0x180d3d]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::main::{{closure}}+0xe) [0x180a4e]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::currentprocess::with::{{closure}}+0x26c) [0x1822ec]
> /home/joerg/git/rustup/target/debug/rustup-init(std::thread::local::LocalKey<T>::try_with+0xdc) [0x181cdc]
> /home/joerg/git/rustup/target/debug/rustup-init(std::thread::local::LocalKey<T>::with+0x31) [0x181bd1]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup::currentprocess::with+0x85) [0x182065]
> /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::main+0x42) [0x180d22]
> /home/joerg/git/rustup/target/debug/rustup-init(core::ops::function::FnOnce::call_once+0xb) [0x18250b]
> /home/joerg/git/rustup/target/debug/rustup-init(std::sys_common::backtrace::__rust_begin_short_backtrace+0xe) [0x183bae]
> /home/joerg/git/rustup/target/debug/rustup-init(std::rt::lang_start::{{closure}}+0x11) [0x183c21]
> /home/joerg/git/rustup/target/debug/rustup-init(std::rt::lang_start_internal+0x441) [0x10c2301]
> /home/joerg/git/rustup/target/debug/rustup-init(std::rt::lang_start+0x3a) [0x183bfa]
> /home/joerg/git/rustup/target/debug/rustup-init(main+0x1e) [0x1816ae]
> /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_init_first+0x8a) [0x276ca]
> /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x27785]
> /home/joerg/git/rustup/target/debug/rustup-init(_start+0x21) [0x1806d1]
error: I/O Error: Too many levels of symbolic links (os error 40)
11:59:57.159481 +++ exited with 1 +++Which points to https://github.com/rust-lang/rustup/blob/0c501d55c54031a992395078d535b6d2574dc693/src/utils/raw.rs#L45-L54 that adds the NOFOLLOW flag
contrary to Toolchain::exists() https://github.com/rust-lang/rustup/blob/0c501d55c54031a992395078d535b6d2574dc693/src/toolchain/toolchain.rs#L70-L74
rami3l
commented
7 months ago @jo-so Wow, many thanks for your help with the investigation! It's true that with libc::O_NOFOLLOW, ELOOP will have a slight different meaning. Thanks again for the reminder!
I have no idea why this option flag is enabled though, and this patch was included before I was even there. Maybe @rbtcollins can provide more context on this one?
Considering the Windows side of the same function, it doesn't seem to have the same semantics:
FILE_FLAG_BACKUP_SEMANTICS 0x02000000 The file is being opened or created for a backup or restore operation. The system ensures that the calling process overrides file security checks when the process has SE_BACKUP_NAME and SE_RESTORE_NAME privileges [..] You must set this flag to obtain a handle to a directory. A directory handle can be passed to some functions instead of a file handle.
OTOH the only caller of this function is Toolchain::exists, and I don't see a particular reason why we can't follow a symlink for this purpose.
rami3l
commented
7 months ago @joso I just did some more research on this one.
The snippet that you mentioned above actually came from remove_dir_all, and the same code copied over to Rustup is indeed the root cause of #3344 😓
fn open_dir(p: &Path) -> io::Result<fs::File> {
let mut options = OpenOptions::new();
options.read(true);
options.custom_flags(libc::O_NOFOLLOW);
options.open(p)
}For the reference, remove_dir_all seems to consistently drop an error when the dir is a symlink (which might not be the expected behavior on our side anyway). However on Windows it generates a custom error, whereas on Unix it returns ELOOP:
fn open_dir(p: &Path) -> Result<File> {
let mut options = OpenOptions::new();
options.read(true);
options.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT);
let maybe_dir = options.open(p)?;
if maybe_dir.metadata()?.is_symlink() {
return Err(io::Error::new(
io::ErrorKind::Other,
"Path is a directory link, not directory",
));
}
Ok(maybe_dir)
}
rbtcollins
commented
7 months ago Indeed thanks for the bug.
@rami3l I think what we need here for .rustup/toolchains and .rustup/downloads, but not for directories in general, are the following semantics:
I think remove_dir_all + the fs_at crates should permit solving this quite nicely.
re remove_dir_all behaviour, see https://docs.rs/remove_dir_all/0.8.2/remove_dir_all/ the last two paragraphs before 'features'. tl;dr symlinks are not directories, and if you're dealing with a symlink, just unlink, don't expect remove_dir_all to delete it.
re: the semantics on windows, the implementation details are different, but roughly identical semantics are achievable - the open with FILE_FLAG_BACKUP_SEMANTICS is required to read the link out of a link, compared to linux where a different syscall is used to read links.
Happy to review a patch.
ijackson
commented
7 months ago Is there a workaround that isn't "copy my 23Gigabyte toolchains directory somewhere the backups will pick itup"?
Currently I am in this state:
BTW: This bug leaves you with a state where you can't do anything with rustup, no update, no compile of any crate (e.g. rustup itself) to debug.
IOW for me this is a very severe regression.
ijackson
commented
7 months ago Is there a workaround that isn't "copy my 23Gigabyte toolchains directory somewhere the backups will pick itup"?
Answering my own question: I was able to get things to work by replacing the symlink and using a bind mount. This is obviously not a long-term solution.
Is there a way I can downgrade my rustup to a non-broken version and prevent it from re-updating itself?
ijackson
commented
7 months ago Answering my own question: I was able to get things to work by replacing the symlink and using a bind mount. This is obviously not a long-term solution.
This gave me a working rustc but rustup is nonfunctional:
error: could not rename component file from '/home/rustcargo/.rustup/tmp/fwn9oio18asynj7x_file' to '/home/rustcargo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/components'
error: could not rename components file from '/home/rustcargo/.rustup/tmp/mif5invkkmwdy5j6_file' to '/home/rustcargo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/components': Invalid cross-device link (os error 18)This is because my .rustup/tmp is also a symlink. It was a link to the same filesystem as toolchains (obviously) but making toolchains a bind mount has broken that.
rbtcollins
commented
7 months ago Is there a way I can downgrade my rustup to a non-broken version and prevent it from re-updating itself?
Yes. To downgrade, download an older version of rustup-init from the dist server https://static.rust-lang.org/rustup and run it to install it.
the env variable RUSTUP_UPDATE_ROOT will override that, so if you set it to an empty web host in your environment, rustup will no longer be able to self-update.
Or, you can pass --no-self-update to rustup when you invoke an update / install. If you're using an IDE that invokes rustup, consult its documentation.
ijackson
commented
6 months ago I tried to follow these instructions. This is FYI and for the information of anyone else with similar troubles.
Yes. To downgrade, download an older version of rustup-init from the dist server https://static.rust-lang.org/rustup and run it to install it.
I was not able to do this because I don't know the precise URL for "an older version of rustup-init", and that server doesn't do directory listings.
the env variable RUSTUP_UPDATE_ROOT will override that, so if you set it to an empty web host in your environment, rustup will no longer be able to self-update.
This doesn't work because rustup bails out on getting a 404 response.
Or, you can pass --no-self-update to rustup when you invoke an update / install. If you're using an IDE that invokes rustup, consult its documentation.
This doesn't work
rustcargo@zealot:~$ rustup --no-self-update toolchain add nightly-2024-04-18
error: unexpected argument '--no-self-update' found
tip: to pass '--no-self-update' as a value, use '-- --no-self-update'
Usage: rustup [OPTIONS] [+toolchain] [COMMAND]
For more information, try '--help'.
rustcargo@zealot:~$ I have worked around things a different way.
rami3l
commented
6 months ago @jo-so @ijackson A new beta release is available, would you mind trying it out?
jo-so
commented
6 months ago It works:
% RUSTUP_UPDATE_ROOT='https://dev-static.rust-lang.org/rustup' rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2024-04-28, rust version 1.79.0-nightly (aed2187d5 2024-04-27)
…
info: checking for self-update
info: downloading self-update
stable-x86_64-unknown-linux-gnu unchanged - rustc 1.77.2 (25ef9e3d8 2024-04-09)
nightly-x86_64-unknown-linux-gnu updated - rustc 1.79.0-nightly (aed2187d5 2024-04-27) (from rustc 1.79.0-nightly (ef8b9dcf2 2024-04-24))
info: cleaning up downloads & tmp directories
% mv .rustup/toolchains .cache/rustup/
% ln -s ../.cache/rustup/toolchains .rustup
% cargo --version
cargo 1.79.0-nightly (b60a15551 2024-04-26)Thanks! Yes, this worked for me.
First, using my existing rustup, but with that rune, and a troublesome symlink at .rustup/downloads:
rustcargo@zealot:~$ RUSTUP_UPDATE_ROOT='https://dev-static.rust-lang.org/rustup' rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2024-05-02, rust version 1.78.0 (9b00956e5 2024-04-29)
info: downloading component 'cargo'
8.0 MiB / 8.0 MiB (100 %) 7.5 MiB/s in 1s ETA: 0s
info: downloading component 'clippy'
[... much output deleted ...]
info: installing component 'rustfmt'
info: checking for self-update
info: downloading self-update
stable-x86_64-unknown-linux-gnu updated - rustc 1.78.0 (9b00956e5 2024-04-29) (from rustc 1.77.2 (25ef9e3d8 2024-04-09))
beta-x86_64-unknown-linux-gnu updated - rustc 1.79.0-beta.3 (f5d04caa7 2024-05-03) (from rustc 1.78.0-beta.8 (13ef05e2b 2024-04-19))
nightly-x86_64-unknown-linux-gnu updated - rustc 1.80.0-nightly (faefc618c 2024-05-07) (from rustc 1.79.0-nightly (7f2fc33da 2024-04-22))
info: cleaning up downloads & tmp directories
thread 'main' panicked at src/utils/utils.rs:616:13:
Unable to clean up /home/rustcargo/.rustup/downloads: Os { code: 40, kind: FilesystemLoop, message: "Too many levels of symbolic links" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
rustcargo@zealot:~$ And, now that it has updated itself:
rustcargo@zealot:~$ RUSTUP_UPDATE_ROOT='https://dev-static.rust-lang.org/rustup' rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'beta-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: checking for self-update
stable-x86_64-unknown-linux-gnu unchanged - rustc 1.78.0 (9b00956e5 2024-04-29)
beta-x86_64-unknown-linux-gnu unchanged - rustc 1.79.0-beta.3 (f5d04caa7 2024-05-03)
nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.80.0-nightly (faefc618c 2024-05-07)
info: cleaning up downloads & tmp directories
rustcargo@zealot:~$
Problem
BTW: This message isn't very helpful.
I'm having a symlink at ~/.rustup/toolchains pointing to ~/.cache/rustup/toolchains
Steps
mkdir ~/rustup-test && cd ~/.rustup && cp -r downloads tmp update-hashes settings.toml ~/rustup-testln -s ~/.rustup/toolchains ~/rustup-test/RUSTUP_HOME=$HOME/rustup-test rustup updatePossible Solution(s)
No response
Notes
This happens since the update from
to
Rustup version
Installed toolchains