Hermit launches linux x86_64 programs in a special, hermetically isolated sandbox to control their execution. Hermit translates normal, nondeterministic behavior, into deterministic, repeatable behavior. This can be used for various applications, including replay-debugging, reproducible artifacts, chaos mode concurrency testing and bug analysis.
Other
1.17k
stars
31
forks
source link
Cargo Build bricks, "Error: future cannot be sent between threads safely", on two different baremetal Linux #48
Describe the bug
A clear and concise description of what the bug is.
On new installations in non-wsl environments, Ubuntu 22.04 and Arch Linux, cargo build breaks with 7 "error: future cannot be sent between threads safely" errors.
This occurs on two different machines, and two different x86_64 operating systems.
Indicate any of these common scenarios that apply:
[ ] a program hangs under hermit
[X] hermit panics internally
[ ] hermit runs the program but divergence (nondeterminism) occurs
To Reproduce
Minimal input to reproduce the behavior.
On baremetal x86_64 linux, git clone, update
Expected behavior
A clear and concise description of what you expected to happen.
Cargo build should pass all building/compiling checks and end.
Environment
[X] Linux kernel version (uname -a): 6.4.10-arch1-1 & Ubuntu 22.04
[ ] CPU version (/proce/cpuinfo):
[ ] Linux distro flavor (/etc/issue, /etc/redhat-release):
Additional context
Attach the logs to this issue as a text file generated by hermit --log=trace --log-file=FOO run.
Add any other context about the problem here.
USER: cargo clean
Removed 2334 files, 1.1GiB total
USER: cargo build
Compiling proc-macro2 v1.0.79
Compiling unicode-ident v1.0.12
Compiling libc v0.2.153
Compiling cfg-if v1.0.0
Compiling autocfg v1.1.0
Compiling serde v1.0.197
Compiling version_check v0.9.4
Compiling memchr v2.7.1
Compiling once_cell v1.19.0
Compiling syn v1.0.109
Compiling bitflags v1.3.2
Compiling pin-project-lite v0.2.13
Compiling parking_lot_core v0.9.9
Compiling smallvec v1.13.1
Compiling libm v0.2.8
Compiling scopeguard v1.2.0
Compiling io-lifetimes v1.0.11
Compiling pin-utils v0.1.0
Compiling lazy_static v1.4.0
Compiling regex-syntax v0.8.2
Compiling futures-sink v0.3.30
Compiling futures-core v0.3.30
Compiling ahash v0.7.8
Compiling crc32fast v1.4.0
Compiling lock_api v0.4.11
Compiling num-traits v0.2.18
Compiling slab v0.4.9
Compiling memoffset v0.6.5
Compiling aho-corasick v1.1.2
Compiling syscalls v0.6.17
Compiling convert_case v0.4.0
Compiling typenum v1.17.0
Compiling adler v1.0.2
Compiling quote v1.0.35
Compiling miniz_oxide v0.7.2
Compiling syn v2.0.52
Compiling futures-channel v0.3.30
Compiling indexmap v1.9.3
Compiling generic-array v0.14.7
Compiling tracing-core v0.1.32
Compiling futures-task v0.3.30
Compiling iana-time-zone v0.1.60
Compiling thiserror v1.0.58
Compiling futures v0.1.31
Compiling futures-io v0.3.30
Compiling flate2 v1.0.28
Compiling proc-macro-error-attr v1.0.4
Compiling getrandom v0.2.12
Compiling mio v0.8.11
Compiling socket2 v0.5.6
Compiling num_cpus v1.16.0
Compiling parking_lot v0.12.1
Compiling signal-hook-registry v1.4.1
Compiling regex-automata v0.4.6
Compiling hashbrown v0.12.3
Compiling rand_core v0.6.4
Compiling nix v0.25.1
Compiling rustix v0.37.27
Compiling byteorder v1.5.0
Compiling static_assertions v1.1.0
Compiling paste v1.0.14
Compiling twox-hash v1.6.3
Compiling proc-macro-error v1.0.4
Compiling serde_json v1.0.114
Compiling rustix v0.36.17
Compiling itoa v1.0.10
Compiling cpp_demangle v0.4.3
Compiling linux-raw-sys v0.3.8
Compiling anyhow v1.0.81
Compiling ppv-lite86 v0.2.17
Compiling unicase v2.7.0
Compiling rand_chacha v0.3.1
Compiling colored v2.1.0
Compiling hex v0.4.3
Compiling async-trait v0.1.77
Compiling unicode-width v0.1.11
Compiling linux-raw-sys v0.1.4
Compiling stable_deref_trait v1.2.0
Compiling procfs v0.15.1
Compiling ryu v1.0.17
Compiling gimli v0.28.1
Compiling rand v0.8.5
Compiling block-buffer v0.10.4
Compiling crypto-common v0.1.6
Compiling regex v1.10.3
Compiling memmap2 v0.5.10
Compiling rustix v0.38.31
Compiling typed-arena v2.0.2
Compiling heck v0.4.1
Compiling os_str_bytes v6.6.1
Compiling terminal_size v0.2.6
Compiling rustc-demangle v0.1.23
Compiling fallible-iterator v0.3.0
Compiling clap_lex v0.2.4
Compiling textwrap v0.16.1
Compiling digest v0.10.7
Compiling object v0.29.0
Compiling atty v0.2.14
Compiling raw-cpuid v10.7.0
Compiling cpufeatures v0.2.12
Compiling strsim v0.10.0
Compiling termcolor v1.4.1
Compiling either v1.10.0
Compiling never-say-never v6.6.666
Compiling linux-raw-sys v0.4.13
Compiling radium v0.3.0
Compiling sha2 v0.10.8
Compiling fastrand v2.0.1
Compiling arrayvec v0.5.2
Compiling bytesize v1.3.0
Compiling shell-words v1.1.0
Compiling diff v0.1.13
Compiling yansi v0.5.1
Compiling pretty v0.12.3
Compiling serde_derive v1.0.197
Compiling serde_repr v0.1.18
Compiling tokio-macros v2.2.0
Compiling futures-macro v0.3.30
Compiling thiserror-impl v1.0.58
Compiling tracing-attributes v0.1.27
Compiling pretty_assertions v1.4.0
Compiling rand_distr v0.4.3
Compiling log v0.4.21
Compiling pkg-config v0.3.30
Compiling regex-syntax v0.6.29
Compiling derive_more v0.99.17
Compiling clap_derive v3.2.25
Compiling powerfmt v0.2.0
Compiling time-core v0.1.2
Compiling overload v0.1.1
Compiling deranged v0.3.11
Compiling num-conv v0.1.0
Compiling futures-util v0.3.30
Compiling num_threads v0.1.7
Compiling unwind-sys v0.1.4
Compiling nu-ansi-term v0.46.0
Compiling tracing-log v0.2.0
Compiling scroll_derive v0.11.1
Compiling foreign-types-macros v0.2.3
Compiling sharded-slab v0.1.7
Compiling thread_local v1.1.8
Compiling time v0.3.34
Compiling crossbeam-utils v0.8.19
Compiling tracing v0.1.40
Compiling unwind v0.4.2
Compiling regex-automata v0.1.10
Compiling foreign-types-shared v0.3.1
Compiling scroll v0.11.0
Compiling env_logger v0.8.4
Compiling foreign-types v0.5.0
Compiling plain v0.2.3
Compiling goblin v0.5.4
Compiling fbinit_macros v0.1.2 (https://github.com/facebookexperimental/rust-shed.git?branch=main#4a0b610c)
Compiling matchers v0.1.0
Compiling quickcheck v1.0.3
Compiling perf-event-open-sys v4.0.0
Compiling iced-x86 v1.21.0
Compiling clap v3.2.25
Compiling crossbeam-channel v0.5.12
Compiling dirs-sys v0.3.7
Compiling futures-executor v0.3.30
Compiling fbinit v0.1.2 (https://github.com/facebookexperimental/rust-shed.git?branch=main#4a0b610c)
Compiling cfg-if v0.1.10
Compiling futures v0.3.30
Compiling sha1_smol v1.0.0
Compiling atomic v0.5.3
Compiling dirs v2.0.2
Compiling test-allocator v0.0.0 (/home/kaiju/Documents/hermit/common/test-allocator)
Compiling similar v2.4.0
Compiling hermetic_infra_hermit_flaky-tests v0.0.0 (/home/kaiju/Documents/hermit/flaky-tests)
Compiling ruzstd v0.5.0
Compiling object v0.32.2
Compiling bitflags v2.4.2
Compiling bytes v1.5.0
Compiling chrono v0.4.35
Compiling bincode v1.3.3
Compiling linked-hash-map v0.5.6
Compiling bitvec v0.17.4
Compiling rand_pcg v0.3.1
Compiling digest v0.0.0 (/home/kaiju/Documents/hermit/common/digest)
Compiling tracing-serde v0.1.3
Compiling edit-distance v0.0.0 (/home/kaiju/Documents/hermit/common/edit-distance)
Compiling uuid v1.7.0
Compiling tokio v1.36.0
Compiling reverie-memory v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
Compiling reverie-syscalls v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
Compiling addr2line v0.21.0
Compiling tracing-subscriber v0.3.18
Compiling tempfile v3.10.1
Compiling tracing-appender v0.2.3
Compiling reverie-process v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
Compiling tokio-util v0.7.10
Compiling tokio-stream v0.1.14
Compiling reverie v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
Compiling detcore-model v0.0.0 (/home/kaiju/Documents/hermit/detcore-model)
Compiling safeptrace v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
warning: the item `FromIterator` is imported redundantly
--> detcore-model/src/collections/replay_cursor.rs:11:5
|
11 | use std::iter::FromIterator;
| ^^^^^^^^^^^^^^^^^^^^^^^
--> /rustc/3cbb93223f33024db464a4df27a13c7cce870173/library/std/src/prelude/mod.rs:129:13
|
= note: the item `FromIterator` is already defined here
|
= note: `#[warn(unused_imports)]` on by default
Compiling reverie-ptrace v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
Compiling detcore v0.0.0 (/home/kaiju/Documents/hermit/detcore)
warning: `detcore-model` (lib) generated 1 warning
warning: the item `RawFd` is imported redundantly
--> detcore/src/fd.rs:23:5
|
23 | use crate::types::RawFd;
| ^^^^^^^^^^^^^^^^^^^
24 | use crate::types::*;
| --------------- the item `RawFd` is already imported here
|
= note: `#[warn(unused_imports)]` on by default
warning: the item `clap` is imported redundantly
--> detcore/src/logdiff.rs:23:5
|
23 | use clap;
| ^^^^ the item `clap` is already defined by prelude
warning: the item `FromIterator` is imported redundantly
--> detcore/src/preemptions.rs:13:5
|
13 | use std::iter::FromIterator;
| ^^^^^^^^^^^^^^^^^^^^^^^
--> /rustc/3cbb93223f33024db464a4df27a13c7cce870173/library/std/src/prelude/mod.rs:129:13
|
= note: the item `FromIterator` is already defined here
warning: the item `SigWrapper` is imported redundantly
--> detcore/src/lib.rs:104:5
|
94 | use types::*;
| -------- the item `SigWrapper` is already imported here
...
104 | use crate::types::SigWrapper;
| ^^^^^^^^^^^^^^^^^^^^^^^^
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:607:37
|
607 | ) -> Result<CpuIdResult, Errno> {
| _____________________________________^
608 | | trace!("handle_cpuid_event: eax: {}, ecx: {}", eax, ecx);
609 | | self.pre_handler_hook(guest, false).await;
610 | | let res = if self.cfg.virtualize_cpuid {
... |
641 | | Ok(res)
642 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:607:37: 642:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:607:37: 642:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<CpuIdResult, reverie::Errno>> + std::marker::Send>>`
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:648:37
|
648 | ) -> Result<RdtscResult, Errno> {
| _____________________________________^
649 | | trace!("handle_rdtsc_event: {:?}", request);
650 | | self.pre_handler_hook(guest, false).await;
651 | | let result = if guest.config().virtualize_time {
... |
688 | | result
689 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:648:37: 689:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:648:37: 689:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<RdtscResult, reverie::Errno>> + std::marker::Send>>`
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:696:40
|
696 | ) -> Result<Option<Signal>, Errno> {
| ________________________________________^
697 | | if signal == Signal::SIGINT && self.cfg.sigint_instakill {
698 | | warn!("Fatal: Exiting hermit container immediately upon SIGINT");
699 | | unrecoverable_shutdown(guest).await
... |
741 | | }
742 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:696:40: 742:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:696:40: 742:6}>>` to `Pin<Box<dyn Future<Output = Result<Option<Signal>, Errno>> + Send>>`
= note: the full name for the type has been written to '/home/kaiju/Documents/hermit/target/debug/deps/detcore-392b287cb643b2af.long-type-3993446771905929726.txt'
= note: consider using `--verbose` to print the full type name to the console
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:812:93
|
812 | async fn handle_thread_start<G: Guest<Self>>(&self, guest: &mut G) -> Result<(), Error> {
| _____________________________________________________________________________________________^
813 | | let detpid = DetPid::from_raw(guest.pid().into());
814 | | trace!(
815 | | "[tid {}] detcore handle_thread_start, pid={}",
... |
857 | | Ok(())
858 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:812:93: 858:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:812:93: 858:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<(), reverie::Error>> + std::marker::Send>>`
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:860:90
|
860 | async fn handle_post_exec<G: Guest<Self>>(&self, guest: &mut G) -> Result<(), Errno> {
| __________________________________________________________________________________________^
861 | | guest.thread_state_mut().past_global_first_execve = true;
862 | | self.pre_handler_hook(guest, false).await;
863 | |
... |
878 | | Ok(())
879 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:860:90: 879:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:860:90: 879:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<(), reverie::Errno>> + std::marker::Send>>`
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:882:71
|
882 | async fn handle_timer_event<G: Guest<Self>>(&self, guest: &mut G) {
| _______________________________________________________________________^
883 | | info!(
884 | | "[detcore, dtid {}] inbound timer preemption event",
885 | | guest.thread_state().dettid
... |
917 | | self.post_handler_hook(guest).await;
918 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:882:71: 918:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:882:71: 918:6}>>` to `Pin<Box<dyn futures::Future<Output = ()> + std::marker::Send>>`
error: future cannot be sent between threads safely
--> detcore/src/lib.rs:924:29
|
924 | ) -> Result<i64, Error> {
| _____________________________^
925 | | self.pre_handler_hook(guest, false).await;
926 | |
927 | | let dettid = guest.thread_state().dettid;
... |
1161 | | res
1162 | | }
| |_____^ future created by async block is not `Send`
|
= help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:924:29: 1162:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
--> detcore/src/lib.rs:244:30
|
241 | "(pre) registers [dtid {}][rcbs {}]. {}",
| -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 | guest.regs().await.display()
| ^^^^^ await occurs here, with `{}` maybe used later
= note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:924:29: 1162:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<i64, reverie::Error>> + std::marker::Send>>`
warning: `detcore` (lib) generated 4 warnings
error: could not compile `detcore` (lib) due to 7 previous errors; 4 warnings emitted
USER: rustc --version
rustc 1.78.0-nightly (3cbb93223 2024-03-13)
Describe the bug A clear and concise description of what the bug is. On new installations in non-wsl environments, Ubuntu 22.04 and Arch Linux, cargo build breaks with 7 "error: future cannot be sent between threads safely" errors. This occurs on two different machines, and two different x86_64 operating systems.
Indicate any of these common scenarios that apply:
To Reproduce Minimal input to reproduce the behavior. On baremetal x86_64 linux, git clone, update
Expected behavior A clear and concise description of what you expected to happen. Cargo build should pass all building/compiling checks and end.
Environment
uname -a): 6.4.10-arch1-1 & Ubuntu 22.04/proce/cpuinfo):/etc/issue,/etc/redhat-release):Additional context Attach the logs to this issue as a text file generated by
hermit --log=trace --log-file=FOO run.Add any other context about the problem here.