Investigate VersionVec assertion failures

ibraheemdev commented 2 years ago

I've been running into spurious failures of the form:

thread 'a_test' panicked at 'assertion failed: `(left != right)`
  left: `VersionVec { versions: [14, 0, 0, 0] }`,
 right: `VersionVec { versions: [14, 0, 0, 0] }`', .cargo/registry/src/github.com-1ecc6299db9ec823/loom-0.5.4/src/rt/atomic.rs:805:17
stack backtrace:
   0: rust_begin_unwind
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
   1: core::panicking::panic_fmt
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
   4: loom::rt::atomic::State::match_rmw_to_stores
   5: scoped_tls::ScopedKey<T>::with
   6: loom::rt::atomic::Atomic<T>::rmw

When running cargo test --test loom on this file. I don't have a reproducible example other than running the tests in that file a couple times. The issue seems to stem from here: https://github.com/tokio-rs/loom/blob/4352288c89c295c040ff501edfa1eb57534b4366/src/rt/atomic.rs#L752 But there's no documentation indicating why that assertion would fail, or what it means.

Pr0methean commented 8 months ago

I can reliably reproduce this with https://github.com/Pr0methean/shared_buffer_rng/commit/109203438b8a89e246c276b62047313d74efb553 by running

LOOM_LOG=trace \
    LOOM_LOCATION=1 \
    LOOM_CHECKPOINT_INTERVAL=1 \
    LOOM_CHECKPOINT_FILE=my_test.json \
RUSTFLAGS="--cfg loom" RUST_BACKTRACE=1 cargo +stable test --release

Pr0methean commented 8 months ago

When running without --release I get an overflow panic instead, which tells me the cause is that some element of the VersionVec is overflowing a u16 and wrapping back around.

thread 'tests::loom_test_at_most_once_delivery' panicked at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/vv.rs:32:9:
attempt to add with overflow
stack backtrace:
   0: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:127:5
   3: loom::rt::vv::VersionVec::inc
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/vv.rs:32:9
   4: loom::rt::thread::Set::active_causality_inc
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/thread.rs:297:9
   5: loom::rt::synchronize::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/mod.rs:142:9
   6: loom::rt::scheduler::Scheduler::with_execution::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/scheduler.rs:43:34
   7: loom::rt::scheduler::Scheduler::with_state::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/scheduler.rs:131:28
   8: scoped_tls::ScopedKey<T>::with
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/scoped-tls-1.0.1/src/lib.rs:171:13
   9: loom::rt::scheduler::Scheduler::with_state
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/scheduler.rs:131:9
  10: loom::rt::scheduler::Scheduler::with_execution
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/scheduler.rs:43:9
  11: loom::rt::execution
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/mod.rs:174:5
  12: loom::rt::synchronize
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/mod.rs:141:5
  13: loom::rt::cell::Cell::start_read
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/cell.rs:57:9
  14: loom::cell::unsafe_cell::UnsafeCell<T>::get
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/cell/unsafe_cell.rs:172:21
  15: shared_buffer_rng::SyncUnsafeCell<T>::with
             at ./src/lib.rs:29:11
  16: shared_buffer_rng::SharedBufferRng<_,_,T>::new::{{closure}}
             at ./src/lib.rs:248:25
  17: loom::thread::spawn_internal::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/thread.rs:168:47
  18: loom::rt::spawn::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/mod.rs:78:13
  19: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  20: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  21: loom::rt::scheduler::spawn_thread::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/loom-0.7.1/src/rt/scheduler.rs:142:17
  22: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  23: generator::gen_impl::GeneratorImpl<A,T>::init_code::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.5/src/gen_impl.rs:336:21
  24: generator::stack::StackBox<F>::call_once
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.5/src/stack/mod.rs:139:13
  25: generator::stack::Func::call_once
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.5/src/stack/mod.rs:121:9
  26: generator::gen_impl::gen_init::{{closure}}
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.5/src/gen_impl.rs:552:9
  27: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  28: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  29: ___rust_try
  30: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: generator::gen_impl::catch_unwind_filter
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.5/src/gen_impl.rs:543:5
  33: generator::gen_impl::gen_init
             at /Users/hennickc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.5/src/gen_impl.rs:570:25
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[ERROR generator::gen_impl] set panic inside generator

tokio-rs / loom

Investigate VersionVec assertion failures #252