search

apache / horaedb

Apache HoraeDB (incubating) is a high-performance, distributed, cloud native time-series database.
https://horaedb.apache.org
Apache License 2.0
2.59k stars 203 forks source link

Drop table panicked using kafka #1543

Closed linanh closed 1 month ago

linanh commented 1 month ago

Describe this problem

Drop table panicked when using kafka, then process exit abnormal. [src/components/panic_ext/src/lib.rs:57](B thread 'horaedb-meta' panicked 'attempt to add with overflow' at "src/wal/src/message_queue_impl/wal.rs:74"

Server version

Version: 2.0.0 Git commit: a1869dc Git branch: main Opt level: 3 Rustc version: 1.77.0-nightly Target: x86_64-unknown-linux-gnu Build date: 2024-06-24T20:26:38.246766624Z

Steps to reproduce

  1. create table demo
  2. drop table demo

Expected behavior

No response

Additional Information

2024-07-15T09:53:10.261198847Z (B2024-07-15 09:53:10.260(B ERRO(B [src/components/panic_ext/src/lib.rs:57](B thread 'horaedb-meta' panicked 'attempt to add with overflow' at "src/wal/src/message_queue_impl/wal.rs:74"
   0: panic_ext::set_panic_hook::{{closure}}
2024-07-15T09:53:10.261297023Z              at horaedb/src/components/panic_ext/src/lib.rs:56:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/alloc/src/boxed.rs:2029:9
      std::panicking::rust_panic_with_hook
2024-07-15T09:53:10.261349283Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:785:13
   2: std::panicking::begin_panic_handler::{{closure}}
2024-07-15T09:53:10.261366843Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:651:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
2024-07-15T09:53:10.261384733Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/sys_common/backtrace.rs:171:18
2024-07-15T09:53:10.261394490Z    4: rust_begin_unwind
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:647:5
2024-07-15T09:53:10.261412093Z    5: core::panicking::panic_fmt
2024-07-15T09:53:10.261421168Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/panicking.rs:72:14
2024-07-15T09:53:10.261430290Z    6: core::panicking::panic
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/panicking.rs:144:5
   7: <wal::message_queue_impl::wal::MessageQueueImpl<M> as wal::manager::WalManager>::mark_delete_entries_up_to::{{closure}}
2024-07-15T09:53:10.261477869Z              at horaedb/src/wal/src/message_queue_impl/wal.rs:74:39
2024-07-15T09:53:10.261486948Z    8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/future/future.rs:124:9
2024-07-15T09:53:10.261522465Z       analytic_engine::instance::drop::Dropper::drop::{{closure}}
             at horaedb/src/analytic_engine/src/instance/drop.rs:75:14
      analytic_engine::instance::engine::<impl analytic_engine::instance::Instance>::drop_table::{{closure}}
2024-07-15T09:53:10.261549821Z              at horaedb/src/analytic_engine/src/instance/engine.rs:392:31
      <analytic_engine::engine::TableEngineImpl as table_engine::engine::TableEngine>::drop_table::{{closure}}
2024-07-15T09:53:10.261580185Z              at horaedb/src/analytic_engine/src/engine.rs:143:67
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/future/future.rs:124:9
2024-07-15T09:53:10.261633371Z       <table_engine::proxy::TableEngineProxy as table_engine::engine::TableEngine>::drop_table::{{closure}}
2024-07-15T09:53:10.261644227Z              at horaedb/src/table_engine/src/proxy.rs:73:71
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
2024-07-15T09:53:10.261661803Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/future/future.rs:124:9
      <catalog_impls::volatile::SchemaImpl as catalog::schema::Schema>::drop_table::{{closure}}
             at horaedb/src/catalog_impls/src/volatile.rs:404:14
2024-07-15T09:53:10.261689160Z   11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/future/future.rs:124:9
      <catalog_impls::cluster_based::SchemaWithCluster as catalog::schema::Schema>::drop_table::{{closure}}
             at horaedb/src/catalog_impls/src/cluster_based.rs:104:49
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/future/future.rs:124:9
      catalog::table_operator::TableOperator::drop_table_on_shard::{{closure}}
             at horaedb/src/catalog/src/table_operator.rs:253:14
2024-07-15T09:53:10.261761369Z       cluster::shard_operator::ShardOperator::drop_table::{{closure}}
             at horaedb/src/cluster/src/shard_operator.rs:318:14
2024-07-15T09:53:10.261793079Z       cluster::shard_set::Shard::drop_table::{{closure}}
             at horaedb/src/cluster/src/shard_set.rs:165:34
2024-07-15T09:53:10.261810271Z       server::grpc::meta_event_service::handle_drop_table_on_shard::{{closure}}
             at horaedb/src/server/src/grpc/meta_event_service/mod.rs:561:10
      server::grpc::meta_event_service::MetaServiceImpl::drop_table_on_shard_internal::{{closure}}::{{closure}}
             at horaedb/src/server/src/grpc/meta_event_service/mod.rs:213:57
2024-07-15T09:53:10.261846429Z   13: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/core.rs:311:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
2024-07-15T09:53:10.261872797Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::Core<T,S>::poll
2024-07-15T09:53:10.261890079Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/core.rs:300:30
2024-07-15T09:53:10.261914775Z       tokio::runtime::task::harness::poll_future::{{closure}}
2024-07-15T09:53:10.261923022Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:476:19
2024-07-15T09:53:10.261931757Z       <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/panic/unwind_safe.rs:272:9
      std::panicking::try::do_call
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:554:40
      std::panicking::try
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:518:19
      std::panic::catch_unwind
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panic.rs:142:14
2024-07-15T09:53:10.261999635Z       tokio::runtime::task::harness::poll_future
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:464:18
2024-07-15T09:53:10.262016676Z       tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:198:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:152:15
2024-07-15T09:53:10.262109800Z       tokio::runtime::task::raw::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/raw.rs:276:5
  14: tokio::runtime::task::raw::RawTask::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/raw.rs:200:18
      tokio::runtime::task::LocalNotified<S>::run
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/mod.rs:400:9
2024-07-15T09:53:10.262164737Z       tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:576:18
2024-07-15T09:53:10.262182747Z       tokio::runtime::coop::with_budget
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73:5
2024-07-15T09:53:10.262217114Z       tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:575:9
2024-07-15T09:53:10.262234767Z   15: tokio::runtime::scheduler::multi_thread::worker::Context::run
2024-07-15T09:53:10.262243415Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:526:24
      tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:491:21
      tokio::runtime::context::scoped::Scoped<T>::set
2024-07-15T09:53:10.262279188Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/scoped.rs:40:9
2024-07-15T09:53:10.262288320Z       tokio::runtime::context::set_scheduler::{{closure}}
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context.rs:176:26
2024-07-15T09:53:10.262312118Z       std::thread::local::LocalKey<T>::try_with
2024-07-15T09:53:10.262323192Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/thread/local.rs:286:16
      std::thread::local::LocalKey<T>::with
2024-07-15T09:53:10.262339545Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/thread/local.rs:262:9
      tokio::runtime::context::set_scheduler
2024-07-15T09:53:10.262356079Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context.rs:176:17
      tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
2024-07-15T09:53:10.262386506Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:486:9
      tokio::runtime::context::runtime::enter_runtime
2024-07-15T09:53:10.262406000Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65:16
2024-07-15T09:53:10.262414668Z       tokio::runtime::scheduler::multi_thread::worker::run
2024-07-15T09:53:10.262422387Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:478:5
  16: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
2024-07-15T09:53:10.262441369Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/worker.rs:447:45
      <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
2024-07-15T09:53:10.262458585Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/blocking/task.rs:42:21
2024-07-15T09:53:10.262467055Z       tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/core.rs:311:17
2024-07-15T09:53:10.262485995Z       tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/loom/std/unsafe_cell.rs:14:9
2024-07-15T09:53:10.262504065Z       tokio::runtime::task::core::Core<T,S>::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/core.rs:300:30
2024-07-15T09:53:10.262523081Z       tokio::runtime::task::harness::poll_future::{{closure}}
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:476:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
2024-07-15T09:53:10.262560526Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/panic/unwind_safe.rs:272:9
      std::panicking::try::do_call
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:554:40
2024-07-15T09:53:10.262583184Z       std::panicking::try
2024-07-15T09:53:10.262588688Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:518:19
      std::panic::catch_unwind
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panic.rs:142:14
      tokio::runtime::task::harness::poll_future
2024-07-15T09:53:10.262611593Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:464:18
2024-07-15T09:53:10.262617665Z       tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:198:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/harness.rs:152:15
2024-07-15T09:53:10.262641383Z       tokio::runtime::task::raw::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/raw.rs:276:5
2024-07-15T09:53:10.262653428Z   17: tokio::runtime::task::raw::RawTask::poll
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/raw.rs:200:18
2024-07-15T09:53:10.262664997Z       tokio::runtime::task::UnownedTask<S>::run
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/task/mod.rs:437:9
2024-07-15T09:53:10.262676809Z       tokio::runtime::blocking::pool::Task::run
             at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/blocking/pool.rs:159:9
      tokio::runtime::blocking::pool::Inner::run
2024-07-15T09:53:10.262694973Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/blocking/pool.rs:513:17
      tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
2024-07-15T09:53:10.262706902Z              at usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/blocking/pool.rs:471:13
      std::sys_common::backtrace::__rust_begin_short_backtrace
2024-07-15T09:53:10.262722948Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/sys_common/backtrace.rs:155:18
  18: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/thread/mod.rs:529:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/panic/unwind_safe.rs:272:9
2024-07-15T09:53:10.262752844Z       std::panicking::try::do_call
2024-07-15T09:53:10.262761324Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:554:40
2024-07-15T09:53:10.262771182Z       std::panicking::try
2024-07-15T09:53:10.262779980Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panicking.rs:518:19
2024-07-15T09:53:10.262788784Z       std::panic::catch_unwind
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/panic.rs:142:14
2024-07-15T09:53:10.262809251Z       std::thread::Builder::spawn_unchecked_::{{closure}}
2024-07-15T09:53:10.262818841Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/thread/mod.rs:528:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/core/src/ops/function.rs:250:5
  19: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
2024-07-15T09:53:10.262857926Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/alloc/src/boxed.rs:2015:9
2024-07-15T09:53:10.262873992Z       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/alloc/src/boxed.rs:2015:9
      std::sys::pal::unix::thread::Thread::new::thread_start
2024-07-15T09:53:10.262913882Z              at rustc/6b4f1c5e782c72a047a23e922decd33e7d462345/library/std/src/sys/pal/unix/thread.rs:108:17
2024-07-15T09:53:10.262923813Z   20: start_thread
2024-07-15T09:53:10.262935140Z              at build/glibc-LcI20x/glibc-2.31/nptl/pthread_create.c:477:8
2024-07-15T09:53:10.262945997Z   21: clone
             at build/glibc-LcI20x/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
2024-07-15T09:53:10.262963620Z (B
Rachelint commented 1 month ago

Thanks for reply, debugging about it.

chunshao90 commented 3 weeks ago

1550 has fixed this bug, you can recheck it. @linanh