search

facebookexperimental / hermit

Hermit launches linux x86_64 programs in a special, hermetically isolated sandbox to control their execution. Hermit translates normal, nondeterministic behavior, into deterministic, repeatable behavior. This can be used for various applications, including replay-debugging, reproducible artifacts, chaos mode concurrency testing and bug analysis.
Other
1.17k stars 31 forks source link

Misaligned Pointer Dereference with every "hermit run" #41

Closed NicholasTroutman closed 10 months ago

NicholasTroutman commented 1 year ago

Describe the bug A clear and concise description of what the bug is.

Any hermit run <target> fails with a `misaligned pointer dereference: address must be a multiple of 0x8 but is ..." The exact address differs slightly, but it must always be a multiple of 0x8, and is not.

Indicate any of these common scenarios that apply:

To Reproduce Minimal input to reproduce the behavior.

hermit run ./rand.py

Expected behavior A clear and concise description of what you expected to happen. rand.py should execute with a deterministic string of numbers.

Environment

Additional context Attach the logs to this issue as a text file generated by hermit --log=trace --log-file=hermit.log run ./rand.py.

hermit.log

bjoernd commented 1 year ago

Seeing the same issue.

NicholasTroutman commented 1 year ago

Building with cargo build --release the error goes away, but doesn't solve the underlying issue.

bjoernd commented 1 year ago

Digging a bit deeper, the error according to the logs is here: https://github.com/facebookexperimental/reverie/blob/main/safeptrace/src/memory.rs#L171

This is essentially a wrapper for PTRACE_POKE and indeed, the definition of the interface says that pointers going into PTRACE_PEEK and PTRACE_POKE should be machine-word aligned (see man ptrace):

   PTRACE_POKEUSER
         Copy the word data to offset addr in the tracee's USER area.  As for PTRACE_PEEKUSER, 
         the offset must typically be word-aligned.  In order to maintain the integrity of the kernel,
         some modifications to the USER  area are disallowed.

Now apparently, Rust didn't mind too much about this until 1.70, when this was transformed into an actual alignment failure and this is what we're seeing here.

I confirmed this by building Hermit with nightly builds from the past months:

nightly-2023-03-01-x86_64-unknown-linux-gnu -> works
nightly-2023-04-01-x86_64-unknown-linux-gnu -> works
nightly-2023-04-03-x86_64-unknown-linux-gnu -> works
nightly-2023-04-04-x86_64-unknown-linux-gnu -> works
nightly-2023-04-05-x86_64-unknown-linux-gnu -> FAIL
nightly-2023-04-07-x86_64-unknown-linux-gnu -> FAIL
nightly-2023-04-15-x86_64-unknown-linux-gnu -> FAIL
nightly-2023-04-20-x86_64-unknown-linux-gnu -> FAIL
bjoernd commented 1 year ago

I now have a working build with stable Rust, but it's getting weirder.

I tried looking at Reverie because this is where the failure comes from. The Reverie main branch right now is at commit

commit bec52bdd29cf635c6545ad040305d2bcbf362533 (HEAD -> main, origin/main, origin/HEAD)
Author: Dimitris Iliopoulos <diliopoulos@meta.com>
Date:   Mon Jul 31 19:46:59 2023 -0700

    migrate affected rust-toolchain.toml files to rust/llvm-fb-15

and indeed, when I run cargo build in my Hermit checkout, I see that exactly this version is pulled via the dependency. Cargo.lock has

[[package]]
name = "reverie-memory"
version = "0.1.0"
source = "git+https://github.com/facebookexperimental/reverie.git?branch=main#bec52bdd29cf635c6545ad040305d2bcbf362533"
[..]

And after that cargo build I end up in the situation described in this issue.

$ ./target/debug/hermit run ./examples/date.sh
WARNING: --preemption-timout requires hardware perf counters which is not supported on this host, resetting preemption-timeout to 0
thread 'main' panicked at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/bec52bd/safeptrace/src/memory.rs:171:34:
misaligned pointer dereference: address must be a multiple of 0x8 but is 0x5646c7cce221
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread caused non-unwinding panic. aborting.
Error: Sandbox container exited unexpectedly
     > Process exited with code: Signaled(SIGSEGV, true)

Now I checked out Reverie locally on my machine and adjusted the various Cargo.toml files in Hermit to point to my local Reverie checkout:

diff --git a/detcore-model/Cargo.toml b/detcore-model/Cargo.toml
index 97bdafc..44796c0 100644
--- a/detcore-model/Cargo.toml
+++ b/detcore-model/Cargo.toml
@@ -13,7 +13,8 @@ chrono = { version = "0.4", features = ["clock", "serde", "std"], default-featur
 clap = { version = "3.2.25", features = ["derive", "env", "regex", "unicode", "wrap_help"] }
 libc = "0.2.139"
 nix = "0.25"
-reverie-syscalls = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie-syscalls = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-syscalls = { path = "../../reverie/reverie-syscalls" }
 serde = { version = "1.0.176", features = ["derive", "rc"] }
 shell-words = "1.1.0"
 tracing = "0.1.35"
diff --git a/detcore/Cargo.toml b/detcore/Cargo.toml
index 02298c7..a8a3f5a 100644
--- a/detcore/Cargo.toml
+++ b/detcore/Cargo.toml
@@ -45,7 +45,8 @@ rand_distr = "0.4"
 rand_pcg = { version = "0.3", features = ["serde1"] }
 raw-cpuid = "10.6.0"
 regex = "1.9.2"
-reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie = { path = "../../reverie/reverie" }
 serde = { version = "1.0.176", features = ["derive", "rc"] }
 serde_json = { version = "1.0.100", features = ["float_roundtrip", "unbounded_depth"] }
 tempfile = "3.5"
@@ -54,6 +55,7 @@ tracing = "0.1.35"

 [dev-dependencies]
 detcore-testutils = { version = "0.0.0", path = "tests/testutils" }
-reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-ptrace = { path = "../../reverie/reverie-ptrace" }
 test-allocator = { version = "0.0.0", path = "../common/test-allocator" }
 test-case = "3.1"
diff --git a/detcore/tests/testutils/Cargo.toml b/detcore/tests/testutils/Cargo.toml
index c12419c..abf9d7f 100644
--- a/detcore/tests/testutils/Cargo.toml
+++ b/detcore/tests/testutils/Cargo.toml
@@ -9,8 +9,10 @@ edition = "2021"
 detcore = { version = "0.0.0", path = "../.." }
 lazy_static = "1.4"
 pretty_assertions = { version = "1.2", features = ["alloc"], default-features = false }
-reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
-reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie = { path = "../../../../reverie/reverie" }
+#reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-ptrace = { path = "../../../../reverie/reverie-ptrace" }
 test-allocator = { version = "0.0.0", path = "../../../common/test-allocator" }
 tokio = { version = "1.29.1", features = ["full", "test-util", "tracing"] }
 tracing = "0.1.35"
diff --git a/hermit-cli/Cargo.toml b/hermit-cli/Cargo.toml
index d218795..227994e 100644
--- a/hermit-cli/Cargo.toml
+++ b/hermit-cli/Cargo.toml
@@ -25,8 +25,10 @@ pretty_assertions = { version = "1.2", features = ["alloc"], default-features =
 rand = { version = "0.8", features = ["small_rng"] }
 rand_pcg = { version = "0.3", features = ["serde1"] }
 regex = "1.9.2"
-reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
-reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie = { path = "../../reverie/reverie" }
+#reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-ptrace = { path = "../../reverie/reverie-ptrace" }
 serde = { version = "1.0.176", features = ["derive", "rc"] }
 serde_json = { version = "1.0.100", features = ["float_roundtrip", "unbounded_depth"] }
 shell-words = "1.1.0"

The local Reverie checkout is at exactly the same commit as the Github repo.

Now, when I run cargo clean; cargo build in Hermit, I am getting:

$ ./target/debug/hermit run ./examples/date.sh
WARNING: --preemption-timout requires hardware perf counters which is not supported on this host, resetting preemption-timeout to 0
2023-08-15T21:07:51.436511Z  WARN reverie_ptrace::task: Unable to intercept CPUID: Underlying hardware does not support CPUID faulting
2023-08-15T21:07:51.438151Z  WARN reverie_ptrace::task: Unable to intercept CPUID: Underlying hardware does not support CPUID faulting
2023-08-15T21:07:51.461175Z  WARN reverie_ptrace::task: Unable to intercept CPUID: Underlying hardware does not support CPUID faulting
2022-00-01_00:00:00_517200000

I am utterly at loss what the difference here is. I'm compiling with the exact same version of Reverie as is in Github, just have it checked out locally. I inspected the Cargo.lock file and there is no difference between the two builds, except the fact that the failing build mentions the fact that the Reverie crates were pulled from Github instead of my local disk.

I'm currently guessing that for some reason the unalgined memory address now is properly aligned, but honestly, I have no idea what to look at next.

bjoernd commented 1 year ago

Now staring at the full backtrace. The issue happens early on when pre-initialising the tracee process and patching its VDSO apparently:

 RUST_BACKTRACE=full ./target/debug/hermit run ./examples/date.sh
WARNING: --preemption-timout requires hardware perf counters which is not supported on this host, resetting preemption-timeout to 0
thread 'main' panicked at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/safeptrace/src/memory.rs:171:34:
misaligned pointer dereference: address must be a multiple of 0x8 but is 0x562ad21b1269
stack backtrace:
   0:     0x562ad2148b5c - std::backtrace_rs::backtrace::libunwind::trace::hd28b74870fb29f5e
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x562ad2148b5c - std::backtrace_rs::backtrace::trace_unsynchronized::ha778ba6652f5fff7
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x562ad2148b5c - std::sys_common::backtrace::_print_fmt::h57512da8fd27ebfe
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x562ad2148b5c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h9ff91e3dfaf4de84
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x562ad217739c - core::fmt::rt::Argument::fmt::hb4c9152c9d66f707
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/core/src/fmt/rt.rs:138:9
   5:     0x562ad217739c - core::fmt::write::hca827d819a7788c0
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/core/src/fmt/mod.rs:1094:21
   6:     0x562ad21451de - std::io::Write::write_fmt::hda6839af442363e2
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/io/mod.rs:1714:15
   7:     0x562ad2148944 - std::sys_common::backtrace::_print::h83dbca21f18ac9f0
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x562ad2148944 - std::sys_common::backtrace::print::h50f6064ce0c0ed75
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x562ad214a1ca - std::panicking::panic_hook_with_disk_dump::{{closure}}::habdb4fb696892949
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/panicking.rs:278:22
  10:     0x562ad2149eb7 - std::panicking::panic_hook_with_disk_dump::h9e67e3f11439835d
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/panicking.rs:312:9
  11:     0x562ad214a7cb - std::panicking::default_hook::h557da10ef8867559
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/panicking.rs:239:5
  12:     0x562ad214a7cb - std::panicking::rust_panic_with_hook::h03521a4f77cf14d2
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/panicking.rs:729:13
  13:     0x562ad214a6c7 - std::panicking::begin_panic_handler::{{closure}}::ha8912bac885c0f14
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/panicking.rs:621:13
  14:     0x562ad2149086 - std::sys_common::backtrace::__rust_end_short_backtrace::h4ba480d82605b76d
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/sys_common/backtrace.rs:170:18
  15:     0x562ad214a412 - rust_begin_unwind
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/panicking.rs:617:5
  16:     0x562ad0c78e63 - core::panicking::panic_nounwind_fmt::h68aabef9d1a51c26
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/core/src/panicking.rs:96:14
  17:     0x562ad0c7906b - core::panicking::panic_misaligned_pointer_dereference::ha31ca2b7b628ed73
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/core/src/panicking.rs:175:5
  18:     0x562ad1662c8c - safeptrace::memory::<impl reverie_memory::MemoryAccess for safeptrace::Stopped>::write::hd457808a0e439e20
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/safeptrace/src/memory.rs:171:34
  19:     0x562ad13d2031 - reverie_memory::MemoryAccess::write_exact::h168263f5e458713d
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-memory/src/lib.rs:95:19
  20:     0x562ad0f62b69 - reverie_ptrace::vdso::vdso_patch::{{closure}}::h3d29c2ea5ca8fcc9
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-ptrace/src/vdso.rs:241:13
  21:     0x562ad0fe8728 - reverie_ptrace::task::TracedTask<L>::tracee_preinit::{{closure}}::he783d191549dd6f6
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-ptrace/src/task.rs:780:32
  22:     0x562ad0f4679b - reverie_ptrace::tracer::postspawn::{{closure}}::h512176251c4c07d4
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-ptrace/src/tracer.rs:339:42
  23:     0x562ad0f417ad - reverie_ptrace::tracer::TracerBuilder<T>::spawn::{{closure}}::h142b094173af90c7
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-ptrace/src/tracer.rs:512:91
  24:     0x562ad0dd15e6 - hermit::run::{{closure}}::hb56f543fd9321118
                               at /home/ec2-user/hermit/hermit-cli/src/lib.rs:86:55
  25:     0x562ad118b8fb - <core::pin::Pin<P> as core::future::future::Future>::poll::hceb19f41dcf51743
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/core/src/future/future.rs:125:9
  26:     0x562ad1342c55 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::h93288243638daaeb
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:665:57
  27:     0x562ad1342497 - tokio::runtime::coop::with_budget::h89cb1d5d165e5a05
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/coop.rs:107:5
  28:     0x562ad1342497 - tokio::runtime::coop::budget::h6f3e08843cfde6e5
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/coop.rs:73:5
  29:     0x562ad1342497 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h35ad5cd9c44c249b
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:665:25
  30:     0x562ad1337711 - tokio::runtime::scheduler::current_thread::Context::enter::h7d306ad1b2160d8c
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:410:19
  31:     0x562ad133e69b - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::h0a7d91eda4b66741
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:664:36
  32:     0x562ad133c773 - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::h2d78def921612745
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:743:68
  33:     0x562ad13ba91e - tokio::runtime::context::scoped::Scoped<T>::set::he55804fcab92af39
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/context/scoped.rs:40:9
  34:     0x562ad136d2db - tokio::runtime::context::set_scheduler::{{closure}}::h2693eee37fad56e0
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/context.rs:176:26
  35:     0x562ad134d197 - std::thread::local::LocalKey<T>::try_with::h428b95f4a23583c3
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/thread/local.rs:270:16
  36:     0x562ad134ab3e - std::thread::local::LocalKey<T>::with::h522871681f08cbc8
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/std/src/thread/local.rs:246:9
  37:     0x562ad136d232 - tokio::runtime::context::set_scheduler::haf597d83b5223d3c
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/context.rs:176:9
  38:     0x562ad133bb92 - tokio::runtime::scheduler::current_thread::CoreGuard::enter::hab059743fb3b0117
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:743:27
  39:     0x562ad133cb6d - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::h6ed4313f1cf8da4a
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:652:19
  40:     0x562ad1334193 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}::h56f0d215fcb16d6c
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:175:28
  41:     0x562ad115dea6 - tokio::runtime::context::runtime::enter_runtime::h16abd1e772ab053a
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/context/runtime.rs:65:16
  42:     0x562ad13330a7 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::hd3f752b098ed87b4
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/scheduler/current_thread/mod.rs:167:9
  43:     0x562ad11811b3 - tokio::runtime::runtime::Runtime::block_on::hbe16532b33e5fc77
                               at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.31.0/src/runtime/runtime.rs:347:47
  44:     0x562ad0dc4c70 - hermit::run::hb21148ec8ef9a872
                               at /home/ec2-user/hermit/hermit-cli/src/lib.rs:90:5
  45:     0x562ad0ce5976 - hermit::run::RunOpts::run_in_container::h8595fd820e20e472
                               at /home/ec2-user/hermit/hermit-cli/src/bin/hermit/run.rs:838:26
  46:     0x562ad0d1829b - hermit::run::RunOpts::run::{{closure}}::h05e623b2c41b7ca1
                               at /home/ec2-user/hermit/hermit-cli/src/bin/hermit/run.rs:632:13
  47:     0x562ad0d5feff - hermit::container::with_container::{{closure}}::h8fe2ffbbbb972bf5
                               at /home/ec2-user/hermit/hermit-cli/src/bin/hermit/container.rs:44:17
  48:     0x562ad0cb7bea - reverie_process::container::Container::run::{{closure}}::{{closure}}::h5cd7fcef40be6406
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-process/src/container.rs:794:68
  49:     0x562ad0d36282 - core::result::Result<T,E>::map::h09ac8a702e5407db
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/core/src/result.rs:746:25
  50:     0x562ad0cb788f - reverie_process::container::Container::run::{{closure}}::hc47310a5084b373e
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-process/src/container.rs:794:29
  51:     0x562ad1aac3c6 - <alloc::boxed::Box<F,A> as core::ops::function::FnMut<Args>>::call_mut::hc2867decfadf0b9f
                               at /rustc/180dffba142c47240ca0d93096ce90b9fd97c8d7/library/alloc/src/boxed.rs:2014:9
  52:     0x562ad1ab19ed - reverie_process::clone::clone_with_stack::callback::he8145f110f7839e8
                               at /home/ec2-user/.cargo/git/checkouts/reverie-9a587e40a0d7d3be/e3c9782/reverie-process/src/clone.rs:29:9
  53:     0x7f873643f314 - __clone
  54:                0x0 - <unknown>
thread caused non-unwinding panic. aborting.
Error: Sandbox container exited unexpectedly
     > Process exited with code: Signaled(SIGSEGV, true)
bjoernd commented 1 year ago

Adding some debugging println!() in the relevant places:

diff --git a/reverie-ptrace/src/vdso.rs b/reverie-ptrace/src/vdso.rs
index 55ad06d..794e14b 100644
--- a/reverie-ptrace/src/vdso.rs
+++ b/reverie-ptrace/src/vdso.rs
@@ -234,10 +234,13 @@ where
             )
             .await?;

+        println!("Iterating VDSO_PATCH_INFO @ {:x}", vdso.address.0);
         for (name, (offset, size, bytes)) in VDSO_PATCH_INFO.iter() {
+            println!("   name {} off {:x} size {}", name, offset, size);
             let start = vdso.address.0 + offset;
             assert!(bytes.len() <= *size);
             let rptr = AddrMut::from_raw(start as usize).unwrap();
+            println!("       rptr: {:?}", rptr);
             memory.write_exact(rptr, bytes)?;
             assert!(*size >= bytes.len());
             if *size > bytes.len() {
diff --git a/safeptrace/src/memory.rs b/safeptrace/src/memory.rs
index 8fe5f70..8ec5350 100644
--- a/safeptrace/src/memory.rs
+++ b/safeptrace/src/memory.rs
@@ -163,6 +163,7 @@ impl MemoryAccess for Stopped {
     }

     fn write(&mut self, addr: AddrMut<u8>, buf: &[u8]) -> Result<usize, Errno> {
+        println!("            write(addr={:?} buf @ {:?})", addr.cast::<u64>(), buf.as_ptr());
         let size = buf.len();
         if size == 0 {
             return Ok(0);

this happens right the first time vdso_patch() tries to patch the VDSO:

Iterating VDSO_PATCH_INFO @ 7ffce2fef000
   name __vdso_time off b90 size 48
       rptr: 0x7ffce2fefb90
            write(addr=0x7ffce2fefb90 buf @ 0x5626b59d55b1)
thread 'main' panicked at /home/ec2-user/.cargo/git/checkouts/reverie-ba666686ce861f1c/897d6e8/safeptrace/src/memory.rs:172:34:
misaligned pointer dereference: address must be a multiple of 0x8 but is 0x5626b59d55b1

whereas with a local build:

Iterating VDSO_PATCH_INFO @ 7ffcfe3b4000
   name __vdso_clock_getres off bd0 size 96
       rptr: 0x7ffcfe3b4bd0
            write(addr=0x7ffcfe3b4bd0 buf @ 0x557ca67457e8)
            write(addr=0x7ffcfe3b4bd8 buf @ 0x557ca80d92b0)
   name __vdso_time off b90 size 48
[..]

From that we at least know that the buffer triggering the assertion is the buf parameter to vdso_patch(), which is the source of the bytes we want to write. As to why this only happens when building exactly the same source from github, I have still no idea. But I guess the solution would be to make sure that buffer is 64-bit aligned as expected by the underlying interface.

bjoernd commented 1 year ago

https://github.com/facebookexperimental/reverie/blob/main/reverie-ptrace/src/vdso.rs#L27 has the byte code that is injected into the VDSO. There is even a comment that this code must be 8 byte aligned. Apparently, it is not in all cases. I'll submit a Reverie PR.

jasonwhite commented 10 months ago

I believe this issue is fixed by https://github.com/facebookexperimental/reverie/pull/22 (thanks to @bjoernd). Closing it out.