search

MrCroxx / RunKV

[WIP] An experimental cloud-native distributed KV engine for OLTP workload.
MIT License
68 stars 7 forks source link

bug: multi version of one key are separated into two sstables in the same level #149

Open MrCroxx opened 2 years ago

MrCroxx commented 2 years ago

When running bench_kv in release mode, found the following panic:

  left: `[]`,
 right: `[118, 52, 51, 57, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48]`', bench/bench_kv/main.rs:206:21
stack backtrace:
thread 'tokio-runtime-worker' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `1`: result: [863288426616, 863288426695]', wheel/src/components/lsm_tree.rs:143:21

Looks like multi versions of one key are separated into two sstables in the same non-overlapping level.

MrCroxx commented 2 years ago

Seems there's also a raft assert failed, maybe related:

Args {
    concurrenty: 1000,
    loop: 100,
    raft_log_store_data_dir: ".run/tmp/bench-kv/raft-log-store-data",
    log_dir: ".run/tmp/bench-kv/log",
}
thread 'tokio-runtime-worker' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `1`: look up key b"k092                                                            0" in level idx: 1, result: [863288427038, 863288426866]', wheel/src/components/lsm_tree.rs:143:21
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panicking.rs:142:14
   2: core::panicking::assert_failed_inner
             at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panicking.rs:218:23
   3: core::panicking::assert_failed
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   6: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  13: runkv_wheel::worker::raft::RaftWorker<RN,F>::handle_ready::{{closure}}::{{closure}}
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  16: tokio::runtime::task::harness::poll_future
  17: tokio::runtime::task::harness::Harness<T,S>::poll
  18: std::thread::local::LocalKey<T>::with
  19: tokio::runtime::thread_pool::worker::Context::run_task
  20: tokio::runtime::thread_pool::worker::Context::run
  21: tokio::macros::scoped_tls::ScopedKey<T>::set
  22: tokio::runtime::thread_pool::worker::run
  23: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  24: tokio::runtime::task::harness::Harness<T,S>::poll
  25: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'tokio-runtime-worker' panicked at 'assertion failed: `(left == right)`
  left: `Some((7751, 2))`,
 right: `None`', /Users/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/f44c574/src/raw_node.rs:501:17
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panicking.rs:142:14
   2: core::panicking::assert_failed_inner
             at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panicking.rs:224:17
   3: core::panicking::assert_failed
   4: runkv_wheel::worker::raft::RaftWorker<RN,F>::handle_ready::{{closure}}::{{closure}}
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   6: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   7: tokio::runtime::task::harness::poll_future
   8: tokio::runtime::task::harness::Harness<T,S>::poll
   9: std::thread::local::LocalKey<T>::with
  10: tokio::runtime::thread_pool::worker::Context::run_task
  11: tokio::runtime::thread_pool::worker::Context::run
  12: tokio::macros::scoped_tls::ScopedKey<T>::set
  13: tokio::runtime::thread_pool::worker::run
  14: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  15: tokio::runtime::task::harness::Harness<T,S>::poll
  16: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.