Open janhohenheim opened 3 months ago
@janhohenheim Can you bisect this to a specific commit?
Also, can you specify your Windows version? Just in case.
@workingjubilee sure, will do later. First, I'm trying to narrow the example a bit down.
For future reference, permalink at that version https://github.com/rust-lang/rust/blob/22a5267c83a3e17f2b763279eb24bb632c45dc6b/library/std/src/sys/sync/once/futex.rs
You should try the nightly too, @ChrisDenton recently did some changes there that may have happened to fix things
@tgross35 thanks for the hint, updated the description :) Nightly is hitting the same thing
Alright, added my Windows system information and linked to a smaller example. Will run cargo-bisect
now.
For your bevy_dylib_crash
repo, I hit what seems to be an intentional panic somewhere else, not sure if this means it's getting past the point where your unreachable!()
would come up or stopping before it
Finished `release` profile [optimized] target(s) in 0.32s
Running `target\release\bevy_quickstart.exe`
thread 'main' panicked at C:\Users\tmgross\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_asset-0.14.0\src\io\source.rs:503:21:
Failed to create file watcher from path "assets", Error { kind: Generic("Input watch path is neither a file nor a directory."), paths: [] }
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/bcf94dec5ba6838e435902120c0384c360126a26/library\std\src\panicking.rs:661
1: core::panicking::panic_fmt
at /rustc/bcf94dec5ba6838e435902120c0384c360126a26/library\core\src\panicking.rs:74
2: bevy_asset::io::source::AssetSource::build
3: bevy_asset::io::source::AssetSourceBuilder::build
4: bevy_asset::io::source::AssetSourceBuilders::build_sources
5: <bevy_asset::AssetPlugin as bevy_app::plugin::Plugin>::build
6: bevy_app::app::App::add_boxed_plugin
7: bevy_app::plugin_group::PluginGroupBuilder::finish
8: main
9: main
10: main
11: std::rt::lang_start_internal
at /rustc/bcf94dec5ba6838e435902120c0384c360126a26/library\std\src\rt.rs:141
12: main
13: invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
14: __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
15: BaseThreadInitThunk
16: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: process didn't exit successfully: `target\release\bevy_quickstart.exe` (exit code: 101)
Just testing with the latest:
rustc 1.81.0-nightly (5affbb171 2024-07-18)
binary: rustc
commit-hash: 5affbb17153bc69a9d5d8d2faa4e399a014a211e
commit-date: 2024-07-18
host: x86_64-pc-windows-msvc
release: 1.81.0-nightly
LLVM version: 18.1.7
Windows 11 23H2 22631.3380
I have a slightly newer version of Windows but otherwise things look the same. Maybe try updating, but not until after the bisect completes :)
(also I textified your system info)
@tgross35 oh sorry, I have "simplified" the example too much. That's my bad. Can you try with https://github.com/TheBevyFlock/bevy_quickstart ? Just running cargo run --release
should produce the described result.
Ah, yeah reproduced with the quickstart
Are you running a bisect, or still trying to minimize? Unfortunate that this takes almost 3 minutes to compile
Relevant bit with RUST_BACKTRACE=full
11: 0x7ffa909b47af - std::sys::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::closure_env$0,never$>
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\sys\backtrace.rs:168
12: 0x7ffa909b7106 - std::panicking::begin_panic_handler
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:665
13: 0x7ffa90a0d6b4 - core::panicking::panic_fmt
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\core\src\panicking.rs:74
14: 0x7ff6d1364176 - _guard_xfg_dispatch_icall_nop
15: 0x7ff6d1366803 - _guard_xfg_dispatch_icall_nop
16: 0x7ff6d133c711 - <bevy_quickstart::game::assets::HandleMap<bevy_quickstart::game::assets::ImageKey> as bevy_ecs::world::FromWorld>::from_world::h409ade5f5f444d1b
17: 0x7ff6d134fd90 - <bevy_quickstart::AppPlugin as bevy_app::plugin::Plugin>::build::h2e1d5ae2b1f18546
18: 0x7ffa44ad6b47 - bevy_app::app::App::add_boxed_plugin::h671372d46ee73030
19: 0x7ff6d135f121 - <bevy_quickstart::AppSet as bevy_ecs::schedule::set::SystemSet>::dyn_hash::h07f7da5483ff379d
20: 0x7ff6d135e4fa - <bevy_quickstart::AppSet as bevy_ecs::schedule::set::SystemSet>::dyn_hash::h07f7da5483ff379d
21: 0x7ff6d135e4e9 - <bevy_quickstart::AppSet as bevy_ecs::schedule::set::SystemSet>::dyn_hash::h07f7da5483ff379d
22: 0x7ffa9099b969 - std::rt::lang_start_internal::closure$2
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\rt.rs:141
23: 0x7ffa9099b969 - std::panicking::try::do_call
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:557
This does not reproduce in debug mode, interesting
@tgross35 minimizing is proving very annoying. I've asked a friend with a pretty fast machine in the meantime to run the bisect.
This does not reproduce in debug mode, interesting
Note that there are some custom settings in the release profile, namely:
[profile.release]
codegen-units = 1
lto = "thin"
opt-level = "s"
strip = "debuginfo"
I do not know if any of these is relevant.
I've not looked at this properly yet but my off the cuff guess would be this is a compiler issue around dylibs. Specifically I would try lto = true
instead of "thin"
.
Some notes off the top of my head:
lto="thin"
.Other things you can do is to try minimizing dependencies and features (e.g. default-features = false
) as well as removing anything else not necessary (e.g. special compiler flags). Still, bevy is a large framework so minimizing may be tricky.
Indeed! Seems like LTO is the difference
Testing some more: this happens with either lto = "thin"
and lto = true
on both stable (1.79.0) and nightly.
@ChrisDenton I've got the information about this failing on stable
from a user (ping @rparrett) but can't verify it on my machine as I can't get the release to build at all on stable
due to the well-known "Too many symbols" issue.
As for bisect: my friend @bash reports that the nightly on 2024-07-17
already fails with a different error:
thread 'main' panicked at /rustc/032be6f7bbe091c7dfa29f115e94b9cc9bae1758\library\std\src\sync\once.rs:217:20:
assertion failed: state_and_queue.addr() & STATE_MASK == RUNNING
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target-bisector-nightly-2024-07-17-x86_64-pc-windows-msvc\release\bevy_quickstart.exe` (exit code: 101)
Interestingly, this is the same error that I got with an entirely different setup: https://github.com/rust-lang/rustc_codegen_cranelift/issues/1508#issuecomment-2209541724
which I assumed to be cranelift
's fault, as I did not get that error using LLVM as a backend in that case.
Testing some more: this happens with either
lto = "thin"
andlto = true
on both stable (1.79.0) and nightly.
Thanks for verifying that this indeed fails on stable.
Ah, it's a very different error on stable. The one in the OP would not be possible.
That futex code isn't used by Windows on stable.
When reporting "other people hit this too", you must determine the exact cause of each failure. It cannot simply be "it failed". You should also NOT list a compiler version if you don't have a confirmed, clean-build on-stable repro. It is quite possible there are 5 different bugs here.
@workingjubilee fair enough, sorry.
I can personally confirm that cargo 1.81.0-nightly
hits this backtrace:
I assumed that futex
was at fault because that is the only place where this error message appears in the repo.
Googling that error message verbatim tells me that the only other place it appears in is a library called rustix-futex-sync, which does not appear in my Cargo.lock
.
If two different implementations (in stable and nightly) are hitting an error then that strongly suggests this is a dylib/lto issue and not an issue with the code itself.
@ChrisDenton
I've got the information about this failing on stable from a user (ping@rparrett
) but can't verify it on my machine as I can't get the release to build at all on stable due to the well-known "Too many symbols" issue.
This was a miscommunication I guess. I never saw this panic, just "too many symbols."
If two different implementations (in stable and nightly) are hitting an error then that strongly suggests this is a dylib/lto issue and not an issue with the code itself.
Looks like so far, the only person able to run into unreachable!
on stable
is @tgross35.
Are you sure it was unreachable!
and not assertion failed: state_and_queue.addr() & STATE_MASK == RUNNING
? I'm asking because that is the error I get when using the nightly
that was published around the time of the last stable
.
I can reproduce a stable error reliably (at least on my machine) with https://github.com/ChrisDenton/bevy_dylab_crash/tree/main
Running `target\release\bevy_quickstart.exe`
thread 'main' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\std\src\sync\once.rs:208:20:
assertion failed: state_and_queue.addr() & STATE_MASK == RUNNING
Didn't read close enough before, but that is indeed the same message I get on stable with https://github.com/TheBevyFlock/bevy_quickstart
Alright, that at least confirms that the unreachable!
execution is indeed only happening on nightly.
About the state_and_queue.addr() & STATE_MASK == RUNNING
: is that kind of error a user error (Bevy in this case) or a standard library bug?
I can reproduce a stable error reliably (at least on my machine) with ChrisDenton/bevy_dylab_crash@
main
Whoa that's a nice minimal example (well more minimal than the template I guess 😆).
I get the original error when I run this with the current nightly (2024-07-18):
Running `target\release\bevy_quickstart.exe`
thread 'main' panicked at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e\library\std\src\sync\once.rs:217:20:
internal error: entered unreachable code: state is never set to invalid values
Cool! Now we've got an example that can toggle between both errors 😄
Yeah, so I would return to my point above. These are using two very different implementations of Once
so it's unlikely that there's a bug in the standard library code. More likely is that there's a dylib or lto (or both) issue that's causing the wrong memory address to be read.
@janhohenheim it's fine. there will be mistakes, but then someone has to state what should be done. and unfortunately with very alarming bugs like these, it is often very hard to determine how to even begin to fix the problem, so being excruciatingly precise can become the minimum.
so, the question arises: is there a previous version of rustc that does make this work?
I was able to reduce @ChrisDenton's example a bit further:
fn main() {
let mut schedule = Schedule::default();
schedule.add_systems(noop);
}
fn noop() {}
@workingjubilee makes sense. I'll remember that next time I open a rustc
issue :)
Unfortunately, going back in time with rustc
versions is very difficult in this case as Bevy 0.14 depends on the newest stable. @bash is currently working on isolating the issue further to make it easier to downgrade the code in question.
huh, this is... weird.
which libraries are being built as dylibs?
Unfortunately, going back in time with rustc versions is very difficult in this case as Bevy 0.14 depends on the newest stable.
I can reproduce at least as far back as 1.76 using bevy 0.13,
huh, this is... weird.
which libraries are being built as dylibs?
The whole dynamic linking business goes through bevy_dylib, which, as far as I understand, pulls in bevy_internal, which in turn pulls in the rest of the Bevy crates.
I was able to create a minimal example that doesn't involve bevy at all! 🎉 https://github.com/bash/rust-dylib-crash
The dylib setup is modeled after that from bevy.
Edit: I'm going to run bisect on this and see what we find out :)
Awesome repro! Interesting that the dylib doesn't even have to do anything, just be there.
In case anyone doesn't want to build & dump themselves:
opt-level=2 to clean things up a bit
Running bisect unattended yields nightly-2022-11-10
as the regressing version.
searched toolchains nightly-2020-01-01 through nightly-2024-07-19
********************************************************************************
Regression in nightly-2022-11-10
********************************************************************************
fetching https://static.rust-lang.org/dist/2022-11-09/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2022-11-09: 40 B / 40 B [===================================================================] 100.00 % 357.72 KB/s converted 2022-11-09 to 85f4f41deb1557ca8ab228319d33003dd2f20f45
fetching https://static.rust-lang.org/dist/2022-11-10/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2022-11-10: 40 B / 40 B [===================================================================] 100.00 % 355.11 KB/s converted 2022-11-10 to e75aab045fc476f176a58c408f6b06f0e275c6e1
looking for regression commit between 2022-11-09 and 2022-11-10
cloning rust repository
opening existing repository at "rust.git"
Found origin remote under name `origin`
refreshing repository at "rust.git"
fetching (via local git) commits from 85f4f41deb1557ca8ab228319d33003dd2f20f45 to e75aab045fc476f176a58c408f6b06f0e275c6e1
opening existing repository at "rust.git"
Found origin remote under name `origin`
refreshing repository at "rust.git"
looking up first commit
looking up second commit
checking that commits are by bors and thus have ci artifacts...
finding bors merge commits
found 9 bors merge commits in the specified range
commit[0] 2022-11-08: Auto merge of #103252 - lcnr:recompute_applicable_impls, r=jackh726
commit[1] 2022-11-08: Auto merge of #104168 - GuillaumeGomez:rollup-tf4edqc, r=GuillaumeGomez
commit[2] 2022-11-09: Auto merge of #103171 - jackh726:gen-interior-hrtb-error, r=cjgillot
commit[3] 2022-11-09: Auto merge of #104179 - Manishearth:rollup-yvsx5hh, r=Manishearth
commit[4] 2022-11-09: Auto merge of #104180 - fee1-dead-contrib:fix-wf-fndef, r=oli-obk
commit[5] 2022-11-09: Auto merge of #102565 - jyn514:refactor-build-manifest, r=Mark-Simulacrum
commit[6] 2022-11-09: Auto merge of #103723 - CastilloDel:master, r=jackh726
commit[7] 2022-11-09: Auto merge of #104192 - Dylan-DPC:rollup-jjo1o80, r=Dylan-DPC
commit[8] 2022-11-09: Auto merge of #104131 - notriddle:notriddle/flate2, r=Mark-Simulacrum
ERROR: no CI builds available between 85f4f41deb1557ca8ab228319d33003dd2f20f45 and e75aab045fc476f176a58c408f6b06f0e275c6e1 within last 167 days
Betting it's in this rollup:
Betting it's this PR:
can someone try this with s/debug_assert/assert/
right there?
can someone try this with
s/debug_assert/assert/
right there?
[building]
edit: grumble I need to update vstools https://users.rust-lang.org/t/error-could-not-compile-rustc-driver-lib-due-to-1-previous-error/113278, feel free to beat me to this if anyone is up to date
can someone try this with
s/debug_assert/assert/
right there?
Just finished. The new assertion doesn't hit (stage1), repro still crashes
@wesleywiser and @michaelwoerister were on the PR, may want to take a look.
It's interesting that the PR causing this behavior was intended to fix a similar bug, https://github.com/rust-lang/rust/issues/81408
I see. The assert is hit but passes. Does reverting the actual change fix this?
Was already building 😆 yes, confirmed that reverting 296489c89268e56abb8f6050842d006b16ed4f09 causes the repro to not crash.
Okay, I think it's fair to say "the way we handle linkage of dylibs when combined with compiler options like LTO results in arriving in what are supposed to be unreachable sections of std's code" is
I tried this code: Clone https://github.com/TheBevyFlock/bevy_quickstart and run
cargo run --release
on WindowsI expected to see this happen: Literally anything else than the standard library hitting unreachable code.
Instead, this happened: We hit this line of unreachable code: https://github.com/rust-lang/rust/blob/22a5267c83a3e17f2b763279eb24bb632c45dc6b/library/std/src/sys/sync/once/futex.rs
Meta
rustc --version --verbose
:Backtrace
``` thread 'main' panicked at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e\library\std\src\sync\once.rs:217:20: internal error: entered unreachable code: state is never set to invalid values stack backtrace: 0: 0x7ffae5f03e3d - std::backtrace_rs::backtrace::dbghelp64::trace at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:91 1: 0x7ffae5f03e3d - std::backtrace_rs::backtrace::trace_unsynchronized at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66 2: 0x7ffae5f03e3d - std::sys::backtrace::_print_fmt at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\sys\backtrace.rs:65 3: 0x7ffae5f03e3d - std::sys::backtrace::impl$0::print::impl$0::fmt at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\sys\backtrace.rs:40 4: 0x7ffae5f34639 - core::fmt::rt::Argument::fmt at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\core\src\fmt\rt.rs:173 5: 0x7ffae5f34639 - core::fmt::write at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\core\src\fmt\mod.rs:1182 6: 0x7ffae5efa941 - std::io::Write::write_fmt
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\io\mod.rs:1827
7: 0x7ffae5f06ed7 - std::panicking::default_hook::closure$1
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:269
8: 0x7ffae5f06ac9 - std::panicking::default_hook
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:296
9: 0x7ffae5f076e2 - std::panicking::rust_panic_with_hook
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:800
10: 0x7ffae5f074ef - std::panicking::begin_panic_handler::closure$0
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:667
11: 0x7ffae5f047af - std::sys::backtrace::__rust_end_short_backtrace
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\sys\backtrace.rs:168
12: 0x7ffae5f07106 - std::panicking::begin_panic_handler
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:665
13: 0x7ffae5f5d6b4 - core::panicking::panic_fmt
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\core\src\panicking.rs:74
14: 0x7ff7dfc60e01 - main
15: 0x7ff7dfc5e391 - main
16: 0x7ff7dfc5cc95 - main
17: 0x7ffade9fe3e2 - bevy_app::app::App::add_boxed_plugin::he4742dca059e0ec1
18: 0x7ff7dfc42aff - main
19: 0x7ff7dfc4104f - main
20: 0x7ff7dfc4103e - main
21: 0x7ffae5eeb969 - std::rt::lang_start_internal::closure$2
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\rt.rs:141
22: 0x7ffae5eeb969 - std::panicking::try::do_call
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:557
23: 0x7ffae5eeb969 - std::panicking::try
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panicking.rs:521
24: 0x7ffae5eeb969 - std::panic::catch_unwind
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\panic.rs:350
25: 0x7ffae5eeb969 - std::rt::lang_start_internal
at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library\std\src\rt.rs:141
26: 0x7ff7dfc42cec - main
27: 0x7ff7dfcd5ba0 - invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
28: 0x7ff7dfcd5ba0 - __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
29: 0x7ffc426a257d - BaseThreadInitThunk
30: 0x7ffc4366af28 - RtlUserThreadStart
error: process didn't exit successfully: `target\release\bevy_quickstart.exe` (exit code: 101)
```