search

tokio-rs / loom

Concurrency permutation testing tool for Rust.
MIT License
2.14k stars 111 forks source link

Checkpointing with Loom #233

Closed matanmarkind closed 3 years ago

matanmarkind commented 3 years ago

I am trying to debug a loom test which is failing, but I am having issues using the checkpoint.

When I run: $ RUST_BACKTRACE=full RUSTFLAGS='--cfg loom' cargo +nightly test --test loom --release

I get: " process didn't exit successfully: . . . (signal: 4, SIGILL: illegal instruction)"

When I try to run with checkpointing I get issues on resolving imports: $ LOOM_CHECKPOINT_FILE=fpath.txt RUST_BACKTRACE=full RUSTFLAGS='--cfg loom --cfg feature="checkpoint"' cargo +nightly test --test loom --release

Compiling generator v0.7.0 Compiling loom v0.5.1 error[E0432]: unresolved import serde --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/object.rs:8:5 | 8 | use serde::{Deserialize, Serialize}; | ^^^^^ use of undeclared crate or module serde

error[E0432]: unresolved import serde --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:4:5 | 4 | use serde::{Deserialize, Serialize}; | ^^^^^ use of undeclared crate or module serde

error[E0432]: unresolved import serde --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/vv.rs:4:5 | 4 | use serde::{Deserialize, Serialize}; | ^^^^^ use of undeclared crate or module serde

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/object.rs:12:43 12 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/object.rs:12:54 12 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/object.rs:36:43 36 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/object.rs:36:54 36 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:8:43 8 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:8:54 8 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:28:43 28 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:28:54 28 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:45:43 45 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:45:54 45 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:58:43 58 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:58:54 58 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:63:47 63 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:63:58 63 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:71:43 71 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/path.rs:71:54 71 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Serialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/vv.rs:9:43 9 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error: cannot determine resolution for the derive macro Deserialize --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/rt/vv.rs:9:54 9 #[cfg_attr(feature = "checkpoint", derive(Serialize, Deserialize))] ^^^^^^^^^^^

= note: import resolution is stuck, try simplifying macro imports

error[E0433]: failed to resolve: use of undeclared crate or module serde_json --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/model.rs:244:9 | 244 | serde_json::from_str(&contents).unwrap() | ^^^^^^^^^^ use of undeclared crate or module serde_json

error[E0433]: failed to resolve: use of undeclared crate or module serde_json --> /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.1/src/model.rs:248:26 | 248 | let serialized = serde_json::to_string(path).unwrap(); | ^^^^^^^^^^ use of undeclared crate or module serde_json

error: aborting due to 23 previous errors

Some errors have detailed explanations: E0432, E0433. For more information about an error, try rustc --explain E0432. error: could not compile loom

To learn more, run the command again with --verbose.

Darksonn commented 3 years ago

I think the problem is that you are enabling the checkpoint feature by passing --cfg feature="checkpoint" in RUSTFLAGS instead of using cargo to enable the feature.

matanmarkind commented 3 years ago

Oh thanks. So it now compiles, but it doesn't seem to output any results to the checkpoint file. I run:

$ LOOM_CHECKPOINT_FILE=my_test.json RUST_BACKTRACE=full RUSTFLAGS='--cfg loom' cargo +nightly test --test="loom" --release --features="loom/checkpoint" lockless_multi
    Finished release [optimized] target(s) in 0.03s
     Running tests/loom.rs (target/release/deps/loom-0ffe8118f46dd66e)

running 1 test
thread panicked while panicking. aborting.
error: test failed, to rerun pass '--test loom'

Caused by:
  process didn't exit successfully: `/home/matan/rust/active_standby/target/release/deps/loom-0ffe8118f46dd66e lockless_multi` (signal: 4, SIGILL: illegal instruction)

This is the same error as without checkpoint, and the file my_test.json doesn't get created. I tried passing an absolute fpath instead with the same result. If I create an empty file of that name the test seems to attempt to read it and files with "EOF while parsing a value". Is there another flag I am missing?

Darksonn commented 3 years ago

Does it fail during the first loom iteration? (e.g. try putting a println! at the top of loom::model)

Darksonn commented 3 years ago

Chances are it's failing before the first checkpoint. Also, you should pass --nocapture to the test.

matanmarkind commented 3 years ago

Cool so adding --nocapture does cause results :) but there is no my_test.json created. (I switched to debug mode so that the trace would be more informative). I suppose this could be something to work from, but I'm curious how to proceed with the checkpointing. Btw if it's easier for you to test out the checkpointing you can just clone the repo that I am using (https://github.com/matanmarkind/active_standby) and experiment.

$ LOOM_CHECKPOINT_FILE=my_test.json RUST_BACKTRACE=full RUSTFLAGS='--cfg loom' cargo +nightly test --test="loom" --features="loom/checkpoint" lockless_multi -- --nocapture
   Compiling active_standby v0.8.5 (/home/matan/rust/active_standby)
    Finished test [unoptimized + debuginfo] target(s) in 2.95s
     Running tests/loom.rs (target/debug/deps/loom-c4e251c9ad869beb)

running 1 test
thread 'loom_tests::lockless_multi_thread' panicked at 'Reached unexpected exploration state. Is the model fully deterministic?', /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/path.rs:166:14
stack backtrace:
   0:     0x55d0ecebffb0 - std::backtrace_rs::backtrace::libunwind::trace::hfa838fc631229987
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x55d0ecebffb0 - std::backtrace_rs::backtrace::trace_unsynchronized::h93a23e36ec026219
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55d0ecebffb0 - std::sys_common::backtrace::_print_fmt::hba56c7f796a4152f
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55d0ecebffb0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h214637f1e26310e1
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x55d0ecee415c - core::fmt::write::h7aa6cd0067dca82a
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/fmt/mod.rs:1110:17
   5:     0x55d0ecebcf45 - std::io::Write::write_fmt::heb07fc0616bbd06d
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/io/mod.rs:1588:15
   6:     0x55d0ecec238b - std::sys_common::backtrace::_print::h2c2441c37e894fb5
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55d0ecec238b - std::sys_common::backtrace::print::h4fb679ac439362ea
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55d0ecec238b - std::panicking::default_hook::{{closure}}::h56bbadec2356e5d2
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:208:50
   9:     0x55d0ecec1e61 - std::panicking::default_hook::hb25822b45f6fdc4e
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:225:9
  10:     0x55d0ecec2a31 - std::panicking::rust_panic_with_hook::h4da5578e7277d2d4
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:622:17
  11:     0x55d0ecec2537 - std::panicking::begin_panic_handler::{{closure}}::h003783ddb3cba4e8
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:519:13
  12:     0x55d0ecec048c - std::sys_common::backtrace::__rust_end_short_backtrace::hd138d2032731ed21
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:141:18
  13:     0x55d0ecec2499 - rust_begin_unwind
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:515:5
  14:     0x55d0ecbb2751 - core::panicking::panic_fmt::hbe99dddd3092ba3c
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/panicking.rs:92:14
  15:     0x55d0ecbb2643 - core::option::expect_failed::h52daa91c8ef676fe
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/option.rs:1243:5
  16:     0x55d0ecc4042f - core::option::Option<T>::expect::h5da537df4b980322
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/option.rs:351:21
  17:     0x55d0ecc36096 - loom::rt::path::Path::branch_load::h5cb9b19d725f0a25
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/path.rs:164:20
  18:     0x55d0ecbe3222 - loom::rt::atomic::Atomic<T>::rmw::{{closure}}::h141a5bacf75dd796
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/atomic.rs:321:25
  19:     0x55d0ecbe57e5 - loom::rt::synchronize::{{closure}}::h472a5db86681a6bb
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:126:19
  20:     0x55d0ecbb9d14 - loom::rt::scheduler::Scheduler::with_execution::{{closure}}::h9686d3b1a5eab05f
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/scheduler.rs:48:28
  21:     0x55d0ecbcdc46 - scoped_tls::ScopedKey<T>::with::h31c998e453530b9c
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-1.0.0/src/lib.rs:171:13
  22:     0x55d0ecbb8b67 - loom::rt::scheduler::Scheduler::with_execution::h8d44537fb0ce0555
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/scheduler.rs:48:9
  23:     0x55d0ecbe9790 - loom::rt::execution::h95968463201ff948
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:157:5
  24:     0x55d0ecbe5240 - loom::rt::synchronize::h5a98369e07e82ba4
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:123:5
  25:     0x55d0ecbe2ff4 - loom::rt::atomic::Atomic<T>::rmw::hefa28c9c97814b79
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/atomic.rs:309:9
  26:     0x55d0ecbb3dcc - loom::sync::atomic::atomic::Atomic<T>::try_rmw::hec2aef3cf8133cb6
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/sync/atomic/atomic.rs:57:9
  27:     0x55d0ecbb3b22 - loom::sync::atomic::atomic::Atomic<T>::compare_exchange::h2fc3164c91835770
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/sync/atomic/atomic.rs:89:9
  28:     0x55d0ecbd2efa - loom::sync::atomic::ptr::AtomicPtr<T>::compare_exchange::h29d661e37233073d
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/sync/atomic/ptr.rs:60:9
  29:     0x55d0ecbd2b73 - active_standby::lockless::table::TableWriteGuard<T>::swap_active_and_standby::h3f258c6dc7aab540
                               at /home/matan/rust/active_standby/src/lockless/table.rs:118:19
  30:     0x55d0ecbde5b5 - <active_standby::lockless::write::WriteGuard<T> as core::ops::drop::Drop>::drop::head9071a73bb4fc4
                               at /home/matan/rust/active_standby/src/lockless/write.rs:241:9
  31:     0x55d0ecbdc09a - core::ptr::drop_in_place<active_standby::lockless::write::WriteGuard<i32>>::h2dcc3d8e73b55331
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  32:     0x55d0ecbda02a - core::ptr::drop_in_place<core::cell::UnsafeCell<active_standby::lockless::write::WriteGuard<i32>>>::h7341b9b5b48f8571
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  33:     0x55d0ecbda547 - core::ptr::drop_in_place<core::option::Option<core::cell::UnsafeCell<active_standby::lockless::write::WriteGuard<i32>>>>::h60214fc98b0e6db8
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  34:     0x55d0ecbde92f - <active_standby::lockless::write::SyncWriteGuard<T> as core::ops::drop::Drop>::drop::h6f0c111e79790f2f
                               at /home/matan/rust/active_standby/src/lockless/write.rs:388:9
  35:     0x55d0ecbdc123 - core::ptr::drop_in_place<active_standby::lockless::write::SyncWriteGuard<i32>>::h3b9502253e5e50ff
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  36:     0x55d0ecbe0e4a - loom::loom_tests::lockless_multi_thread::{{closure}}::{{closure}}::h699bd2dff01acb1e
                               at /home/matan/rust/active_standby/tests/loom.rs:104:13
  37:     0x55d0ecbbe12c - loom::thread::spawn_internal::{{closure}}::h5291f6c3bc88b49a
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/thread.rs:136:47
  38:     0x55d0ecbe952f - loom::rt::spawn::{{closure}}::he0c81a39ae4434e4
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:76:9
  39:     0x55d0ecbd9d5e - core::ops::function::FnOnce::call_once{{vtable.shim}}::he1fef46575cdfa74
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ops/function.rs:227:5
  40:     0x55d0ecc587cd - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::he40229c415fbc4a7
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/alloc/src/boxed.rs:1575:9
  41:     0x55d0ecc8c48e - loom::rt::scheduler::spawn_threads::{{closure}}::{{closure}}::hefe9075c7c19cab2
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/scheduler.rs:140:21
  42:     0x55d0ecc767cd - generator::gen_impl::GeneratorImpl<A,T>::init_code::{{closure}}::h3e7406059f1e4cdc
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/gen_impl.rs:348:21
  43:     0x55d0ecc609db - generator::stack::StackBox<F>::call_once::hcc255fdf157b8427
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/stack/mod.rs:139:13
  44:     0x55d0ece9109b - generator::stack::Func::call_once::hf52e143fc7eb3beb
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/stack/mod.rs:121:9
  45:     0x55d0ece92f43 - generator::gen_impl::gen_init::{{closure}}::h9cd55c896b93bb07
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/gen_impl.rs:550:9
  46:     0x55d0ece92883 - core::ops::function::FnOnce::call_once::h138924f0a4c19689
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ops/function.rs:227:5
  47:     0x55d0ece9561e - std::panicking::try::do_call::h77f0cb84587e3fd9
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:401:40
  48:     0x55d0ece956fd - __rust_try
  49:     0x55d0ece95584 - std::panicking::try::hb4745baf922c9954
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:365:19
  50:     0x55d0ece9279f - std::panic::catch_unwind::h27b901764028d399
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panic.rs:434:14
  51:     0x55d0ece92de4 - generator::gen_impl::gen_init::h32ccb468d0039c30
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/gen_impl.rs:564:25
  52:                0x0 - <unknown>
thread 'loom_tests::lockless_multi_thread' panicked at 'Reached unexpected exploration state. Is the model fully deterministic?', /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/path.rs:274:14
stack backtrace:
   0:     0x55d0ecebffb0 - std::backtrace_rs::backtrace::libunwind::trace::hfa838fc631229987
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x55d0ecebffb0 - std::backtrace_rs::backtrace::trace_unsynchronized::h93a23e36ec026219
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55d0ecebffb0 - std::sys_common::backtrace::_print_fmt::hba56c7f796a4152f
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55d0ecebffb0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h214637f1e26310e1
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x55d0ecee415c - core::fmt::write::h7aa6cd0067dca82a
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/fmt/mod.rs:1110:17
   5:     0x55d0ecebcf45 - std::io::Write::write_fmt::heb07fc0616bbd06d
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/io/mod.rs:1588:15
   6:     0x55d0ecec238b - std::sys_common::backtrace::_print::h2c2441c37e894fb5
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55d0ecec238b - std::sys_common::backtrace::print::h4fb679ac439362ea
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55d0ecec238b - std::panicking::default_hook::{{closure}}::h56bbadec2356e5d2
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:208:50
   9:     0x55d0ecec1e61 - std::panicking::default_hook::hb25822b45f6fdc4e
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:225:9
  10:     0x55d0ecec2a31 - std::panicking::rust_panic_with_hook::h4da5578e7277d2d4
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:622:17
  11:     0x55d0ecec2537 - std::panicking::begin_panic_handler::{{closure}}::h003783ddb3cba4e8
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:519:13
  12:     0x55d0ecec048c - std::sys_common::backtrace::__rust_end_short_backtrace::hd138d2032731ed21
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/sys_common/backtrace.rs:141:18
  13:     0x55d0ecec2499 - rust_begin_unwind
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:515:5
  14:     0x55d0ecbb2751 - core::panicking::panic_fmt::hbe99dddd3092ba3c
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/panicking.rs:92:14
  15:     0x55d0ecbb2643 - core::option::expect_failed::h52daa91c8ef676fe
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/option.rs:1243:5
  16:     0x55d0ecc406cf - core::option::Option<T>::expect::hbf8b789b408fac44
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/option.rs:351:21
  17:     0x55d0ecc36db2 - loom::rt::path::Path::branch_thread::h0663405fe82bce40
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/path.rs:272:24
  18:     0x55d0ecc63dd2 - loom::rt::execution::Execution::schedule::he3ca68de7b547e0a
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/execution.rs:182:20
  19:     0x55d0ecc6715e - loom::rt::branch::{{closure}}::h248e1f83881fd925
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:105:22
  20:     0x55d0ecc8aee6 - loom::rt::scheduler::Scheduler::with_execution::{{closure}}::haa050bc07e4a2964
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/scheduler.rs:48:28
  21:     0x55d0ecc995f9 - scoped_tls::ScopedKey<T>::with::h39a6b4d54c2d58d2
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-1.0.0/src/lib.rs:171:13
  22:     0x55d0ecc891db - loom::rt::scheduler::Scheduler::with_execution::h0401c4e824eb8493
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/scheduler.rs:48:9
  23:     0x55d0ecc6aef4 - loom::rt::execution::hb1b95d2aa4fce401
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:157:5
  24:     0x55d0ecc670c4 - loom::rt::branch::hd5612958d33236b2
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:103:25
  25:     0x55d0ecc79a69 - loom::rt::object::Ref<T>::branch_action::hc328b649c7cfa683
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/object.rs:343:9
  26:     0x55d0ecc2fbd6 - loom::rt::arc::Arc::branch::he10a482f7d146d6d
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/arc.rs:130:9
  27:     0x55d0ecc2f4a1 - loom::rt::arc::Arc::ref_dec::h20b1a420127128da
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/arc.rs:98:9
  28:     0x55d0ecbdd3cd - <loom::sync::arc::Arc<T> as core::ops::drop::Drop>::drop::h7bc8d32c0c1e989f
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/sync/arc.rs:120:12
  29:     0x55d0ecbdaa43 - core::ptr::drop_in_place<loom::sync::arc::Arc<loom::sync::mutex::Mutex<slab::Slab<loom::sync::arc::Arc<loom::sync::atomic::int::AtomicUsize>>>>>::h7f7d6a8fb3da4337
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  30:     0x55d0ecbdbf13 - core::ptr::drop_in_place<active_standby::lockless::write::Writer<i32>>::h9c82465132dc0c3b
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  31:     0x55d0ecbd9fba - core::ptr::drop_in_place<core::cell::UnsafeCell<active_standby::lockless::write::Writer<i32>>>::hd6874d69d639bb8b
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  32:     0x55d0ecbdc053 - core::ptr::drop_in_place<active_standby::lockless::write::SyncWriter<i32>>::h4102d7ad2d6fa90d
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  33:     0x55d0ecbda3da - core::ptr::drop_in_place<loom::loom_tests::lockless_multi_thread::{{closure}}::{{closure}}>::hb89a05d81da01e88
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ptr/mod.rs:192:1
  34:     0x55d0ecbe0dcf - loom::loom_tests::lockless_multi_thread::{{closure}}::{{closure}}::h699bd2dff01acb1e
                               at /home/matan/rust/active_standby/tests/loom.rs:104:13
  35:     0x55d0ecbbe12c - loom::thread::spawn_internal::{{closure}}::h5291f6c3bc88b49a
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/thread.rs:136:47
  36:     0x55d0ecbe952f - loom::rt::spawn::{{closure}}::he0c81a39ae4434e4
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/mod.rs:76:9
  37:     0x55d0ecbd9d5e - core::ops::function::FnOnce::call_once{{vtable.shim}}::he1fef46575cdfa74
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ops/function.rs:227:5
  38:     0x55d0ecc587cd - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::he40229c415fbc4a7
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/alloc/src/boxed.rs:1575:9
  39:     0x55d0ecc8c48e - loom::rt::scheduler::spawn_threads::{{closure}}::{{closure}}::hefe9075c7c19cab2
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.2/src/rt/scheduler.rs:140:21
  40:     0x55d0ecc767cd - generator::gen_impl::GeneratorImpl<A,T>::init_code::{{closure}}::h3e7406059f1e4cdc
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/gen_impl.rs:348:21
  41:     0x55d0ecc609db - generator::stack::StackBox<F>::call_once::hcc255fdf157b8427
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/stack/mod.rs:139:13
  42:     0x55d0ece9109b - generator::stack::Func::call_once::hf52e143fc7eb3beb
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/stack/mod.rs:121:9
  43:     0x55d0ece92f43 - generator::gen_impl::gen_init::{{closure}}::h9cd55c896b93bb07
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/gen_impl.rs:550:9
  44:     0x55d0ece92883 - core::ops::function::FnOnce::call_once::h138924f0a4c19689
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/core/src/ops/function.rs:227:5
  45:     0x55d0ece9561e - std::panicking::try::do_call::h77f0cb84587e3fd9
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:401:40
  46:     0x55d0ece956fd - __rust_try
  47:     0x55d0ece95584 - std::panicking::try::hb4745baf922c9954
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panicking.rs:365:19
  48:     0x55d0ece9279f - std::panic::catch_unwind::h27b901764028d399
                               at /rustc/ed597e7e19d0fe716d9f81b1e840a5abbfd7c28d/library/std/src/panic.rs:434:14
  49:     0x55d0ece92de4 - generator::gen_impl::gen_init::h32ccb468d0039c30
                               at /home/matan/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/gen_impl.rs:564:25
  50:                0x0 - <unknown>
thread panicked while panicking. aborting.
error: test failed, to rerun pass '--test loom'

Caused by:
  process didn't exit successfully: `/home/matan/rust/active_standby/target/debug/deps/loom-c4e251c9ad869beb lockless_multi --nocapture` (signal: 4, SIGILL: illegal instruction)
Darksonn commented 3 years ago

It sounds like your loom model is not deterministic. You'll have to fix that.

As for the checkpoint stuff, the default interval is too large so that it fails before it reaches the checkpoint. You can try to reduce it.

matanmarkind commented 3 years ago

Ok, I thought I needed to generate the my_test.json before using LOOM_CHECKPOINT_INTERVAL. Thanks.

What does it mean that my model is not deterministic/can you point me to resources? Does this refer to a problem in the test (run through loom::model) or does it point to a flaw in the underlying data structures? I thought that being non deterministic would mean that my test would fail because the assertions within it don't hold. Sorry, I'm a bit new to memory modeling and I appreciate your patience :)

Darksonn commented 3 years ago

It means that it doesn't behave in exactly the same way every time you run it. For example, we had one problem of this type in Tokio where we stored JoinHandles in a hash map, then iterated over it which was non-deterministic since the iteration order of a hash map changes from one run to another.

matanmarkind commented 3 years ago

Thanks a lot for your help I've got everything working now!