search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.78k stars 561 forks source link

Compactor errors with duplicate key #18262

Open pjpringlenom opened 2 weeks ago

pjpringlenom commented 2 weeks ago

Describe the bug

Running DDL to drop table hangs Checking await tree can see actors stuck on store_flush Compactor process panics and restarts Recently upgraded from 1.8.2 to 1.10.0

Error message/log

Dropped table_ids: {1815, 1777, 1803, 152, 1793, 149, 1936, 1773, 1778, 1795, 1753, 188, 1933, 173, 1808, 175, 162, 1814, 1776, 1788, 1807, 1798}

thread 'rw-compaction' panicked at /risingwave/src/storage/hummock_sdk/src/key.rs:1089:21:
key UserKey { 204, TableKey { 000000017065727369737465096e743a2f2f707269096d652d7472616469096e672f7072696d65092d706f736974696f096e2d646174612f7309706f742d66782d6a09736f6e000000000003 }
} epoch EpochWithGap(7013100399230976) >= prev epoch EpochWithGap(7013100399230976)
stack backtrace:
2024-08-27T05:04:10.110414959Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=2 pull_task_ack=false pending_pull_task_count=0rust_begin_unwind: 0
5:645:rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs/.             at
panic_fmt::panicking::core: 1
14:72:rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs/.             at
   2: risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe_multi_version
             at ./risingwave/src/storage/hummock_sdk/src/key.rs:1089:21
   3: risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe
             at ./risingwave/src/storage/hummock_sdk/src/key.rs:1038:9
   4: risingwave_storage::hummock::compactor::compactor_runner::compact_and_build_sst::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:662:31
   5: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
   6: risingwave_storage::hummock::compactor::Compactor::compact_key_range_impl::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/mod.rs:262:10
   7: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
   8: risingwave_storage::hummock::compactor::Compactor::compact_key_range::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/mod.rs:160:18
   9: risingwave_storage::hummock::compactor::compactor_runner::CompactorRunner::run::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:152:14
  10: risingwave_storage::hummock::compactor::compactor_runner::compact::{{closure}}::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:489:18
  11: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  12: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  13: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19}}closure{{::instrument::TreeRoot::root::await_tree: 14
34:43:root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/root.rs/.             at
poll::>Future::future::future::core as >B,A<Either::either::future::futures_util<: 15
32:109:root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs/.             at

  16: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once             at
./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9  22: std::
panicking::try::do_call./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552             at
:40: std::panicking::try  23
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs
:516:19 24: std::
panic::catch_unwind             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146
:14: tokio::runtime::  25
task::harness::poll_future             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:
473:18Harness<T,S>::task::harness:: 26: tokio::runtime
::poll_inner/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27             at .
harness::Harness<T,tokio::runtime::task::  27:
S>::poll             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153
:15
  28: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  29: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  30: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:639:22
  31: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  32: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  33: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run

             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  36: tokio::runtime::context::scoped::Scoped<T>::set
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  37: tokio::runtime::context::set_scheduler::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  38: std::thread::local::LocalKey<T>::try_with
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:284:16
  39: std::thread::local::LocalKey<T>::with
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:260:9
  40: tokio::runtime::context::set_scheduler
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17
  41: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  42: tokio::runtime::context::runtime::enter_runtime
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  43: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  44: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  45: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  46: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9  47: tokio::runtime::task::core::Core
<T,S>::poll::{{closure}}./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17             at
UnsafeCell<T>::with_mut::loom::std::unsafe_cell::  48: tokio
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:
16:9task::core::Core<T 49: tokio::runtime::
,S>::poll             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317
:30
  50: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  51: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  52: std::panicking::try::do_call
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  53: std::panicking::try
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  54: std::panic::catch_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  55: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  56: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  57: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  58: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  59: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  60: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  61: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  62: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

*** await tree context of current task ***

Compaction Task 9595335 Split 0  [1.439s]
  compact [1.439s]
    compact_and_build_sst [1.439s]  <== current

To Reproduce

Periodic failures of compaction process looks to be caused by duplicate key check (https://github.com/risingwavelabs/risingwave/issues/15099) Issues completing barriers in streaming jobs these are stuck at store_flush[!!! 123s]

Expected behavior

Streaming actors should not get stuck Compactor should run successfully

How did you deploy RisingWave?

docker compose using v1.10.0

The version of RisingWave

PostgreSQL 13.14.0-RisingWave-1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c)

Additional context

No response

hzxa21 commented 1 week ago

Can you share the following information?

pjpringlenom commented 1 week ago

Looks same as issue https://github.com/risingwavelabs/risingwave/issues/16235

Seems more frequent after running ddl to drop/create tables. For example drop a source config and create a new one with the same name.

drop source/table xyz cascade

create source/table xyz

xxchan commented 1 week ago

Hi @pjpringlenom , could you provide more information as @hzxa21 mentioned above?

Besides,

pjpringlenom commented 1 week ago

Using docker postgres as meta backend Connectors pulsar - no new partitions etc Dropping and recreating sources/tables/mvs

Have a new example from yesterday; logs across, meta, compute x 2 and compactors x 2


select * from rw_internal_tables where id = 10;\
"id","name","schema_id","owner","definition","acl","initialized_at","created_at","initialized_at_cluster_version","created_at_cluster_version"
10,__internal_XXX_table_6_source_0,2,1,"","{root=arwd/root,postgres=arwd/root}",2024-08-29 14:47:34.821 +0800,2024-08-29 14:47:34.821 +0800,PostgreSQL 13.14.0-RisingWave-1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c),PostgreSQL 13.14.0-RisingWave-1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c)

logs around this time

2024-09-05T09:20:59.138118662Z

### meta-node-0

2024-09-05T09:20:48.729774358Z  INFO risingwave_meta::hummock::vacuum: Finish vacuuming SSTs [843484, 843522, 843536, 843475, 84345
6, 843474, 843480, 843531, 843527, 843481, 843538, 843468, 843499, 843548, 843473, 843490, 843497, 843526, 843554, 843452, 843525,
843519, 843540, 843537, 843550, 843541, 843498, 843476, 843470, 843551, 843524, 843539, 843503, 843528, 843483, 843517, 843502, 843
488, 843504, 843491, 843534, 843467, 843492, 843455, 843507, 843505, 843500, 843506, 843495, 843530, 843529, 843549, 843469, 843472
, 843444, 843494, 843486, 843493, 843533, 843535, 843368, 843520, 843478, 843489, 843523, 843482, 843518, 843547, 843485, 843501, 8
43487, 843543, 843521, 843496, 843471, 843457, 843532, 843477, 843479]
2024-09-05T09:20:53.138786708Z  INFO risingwave_meta::hummock::manager::timer_task: Heartbeat expired compaction tasks detected. Attempting to cancel tasks. expired_tasks=[485720]
2024-09-05T09:20:59.969681309Z  WARN risingwave_meta::hummock::manager::compaction: compactor stream 3 poll with err, recv stream may be destroyed error=status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }: error reading a body from connection: stream closed because of a broken pipe
2024-09-05T09:21:00.58572198Z  INFO risingwave_meta_service::cluster_service: resolve host addr socket_addr=10.65.32.194:6660 host=HostAddress { host: "host1", port: 6660 }
2024-09-05T09:21:01.694416719Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 3
2024-09-05T09:21:01.694479024Z  INFO risingwave_meta::hummock::manager::compaction: compactor 3 enters the cluster
2024-09-05T09:21:05.883066907Z  INFO risingwave_meta::hummock::manager::timer_task: Heartbeat expired compaction tasks detected. Attempting to cancel tasks. expired_tasks=[485736, 485735, 485734]
2024-09-05T09:21:07.519870027Z  WARN risingwave_meta::hummock::manager::compaction: compactor stream 9 poll with err, recv stream may be destroyed error=status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }: error reading a body from connection: stream closed because of a broken pipe
2024-09-05T09:21:07.802400412Z  INFO risingwave_meta_service::cluster_service: resolve host addr socket_addr=10.65.37.184:6660 host=HostAddress { host: "host2", port: 6660 }
2024-09-05T09:21:08.854264464Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 9
2024-09-05T09:21:08.854404726Z  INFO risingwave_meta::hummock::manager::compaction: compactor 9 enters the cluster
2024-09-05T09:21:18.838648908Z  INFO risingwave_meta::hummock::vacuum: Finish vacuuming SSTs [843659, 843558, 843579, 843631, 843627, 843611, 843667, 843636, 843556, 843542, 843598, 843584, 843615, 843599, 843571, 843580, 843661, 843607, 843557, 843572, 843570, 843604, 843605, 843544, 843555, 843564, 843593, 843602, 843671, 843672, 843561, 843597, 843603, 843643, 843675, 843574, 843590, 843614, 843585, 843609, 843559, 843610, 843669, 843658, 843633, 843628, 843567, 843582, 843546, 843553, 843639, 843666, 843545, 843638, 843563, 843601, 843566, 843592, 843595, 843635, 843637, 843630, 843576, 843577, 843565, 843596, 843587, 843583, 843552, 843608, 843591, 843581, 843613, 843663, 843578, 843642, 843560, 843573, 843662, 843616, 843586, 843594, 843646, 843640, 843629, 843665, 843589, 843600, 843568, 843612, 843645, 843674, 843678, 843575, 843606, 843679, 843562, 843588, 843632, 843569]
2024-09-05T09:21:30.801197491Z  INFO risingwave_meta::hummock::manager::timer_task: Heartbeat expired compaction tasks detected. Attempting to cancel tasks. expired_tasks=[485770]
2024-09-05T09:21:31.51498701Z  WARN risingwave_meta::hummock::manager::compaction: compactor stream 9 poll with err, recv stream may be destroyed error=status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }: error reading a body from connection: stream closed because of a broken pipe
2024-09-05T09:21:31.778915933Z  INFO risingwave_meta_service::cluster_service: resolve host addr socket_addr=10.65.37.184:6660 host=HostAddress { host: "host2", port: 6660 }
2024-09-05T09:21:32.826223933Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 9
2024-09-05T09:21:32.826395986Z  INFO risingwave_meta::hummock::manager::compaction: compactor 9 enters the cluster
2024-09-05T09:21:38.893304886Z  INFO risingwave_meta::hummock::manager::timer_task: Heartbeat expired compaction tasks detected. Attempting to cancel tasks. expired_tasks=[485793, 485792, 485795]
2024-09-05T09:21:41.578776781Z  WARN risingwave_meta::hummock::manager::compaction: compactor stream 3 poll with err, recv stream may be destroyed error=status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }: error reading a body from connection: stream closed because of a broken pipe
2024-09-05T09:21:42.001652582Z  INFO risingwave_meta_service::cluster_service: resolve host addr socket_addr=10.65.32.194:6660 host=HostAddress { host: "host1", port: 6660 }
2024-09-05T09:21:43.048915206Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 3
2024-09-05T09:21:43.049091286Z  INFO risingwave_meta::hummock::manager::compaction: compactor 3 enters the cluster
2024-09-05T09:21:46.010853272Z  INFO lookup_partitioned_topic_number:lookup_partitioned_topic_number:get_base_connection:get_connection:connect:connect_inner: pulsar::connection_manager: Connected n°6a21ad2a-3562-423d-876b-5287bed55f15 to pulsar://pulsarhost:6850 in 385ms
2024-09-05T09:21:46.01120925Z  INFO lookup_partitioned_topic_number:lookup_partitioned_topic_number:get_base_connection:get_connection:connect:connect_inner: pulsar::connection_manager: Connected n°52ec3e81-07b4-48b6-9c78-d451bda22d84 to pulsar://pulsarhost:6850 in 385ms
2024-09-05T09:21:46.011383107Z  INFO lookup_partitioned_topic_number:lookup_partitioned_topic_number:get_base_connection:get_connection:connect:connect_inner: pulsar::connection_manager: Connected n°f54d362d-ba5f-47d8-9207-946452d4bd3c to pulsar://pulsarhost:6850 in 369ms
2024-09-05T09:21:48.736896598Z  INFO risingwave_meta::hummock::vacuum: Finish vacuuming SSTs [843730, 843692, 843680, 843743, 843727, 843762, 843689, 843690, 843695, 843751, 843715, 843761, 843712, 843754, 843670, 843708, 843747, 843713, 843685, 843704, 843732, 843756, 843714, 843737, 843677, 843706, 843676, 843717, 843691, 843759, 843648, 843699, 843683, 843710, 843733, 843682, 843749, 843701, 843700, 843668, 843757, 843735, 843729, 843716, 843686, 843687, 843738, 843728, 843753, 843740, 843694, 843688, 843664, 843698, 843707, 843681, 843697, 843752, 843745, 843693, 843748, 843657, 843742, 843711, 843702, 843644, 843660, 843696, 843703, 843684, 843649, 843705, 843673, 843634, 843709, 843641, 843746]
2024-09-05T09:22:02.025482719Z  INFO risingwave_meta::hummock::manager::timer_task: Heartbeat expired compaction tasks detected. Attempting to cancel tasks. expired_tasks=[485824]
2024-09-05T09:22:02.938453154Z  WARN risingwave_meta::hummock::manager::compaction: compactor stream 3 poll with err, recv stream may be destroyed error=status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }: error reading a body from connection: stream closed because of a broken pipe
2024-09-05T09:22:03.746559214Z  INFO risingwave_meta_service::cluster_service: resolve host addr socket_addr=10.65.32.194:6660 host=HostAddress { host: "host1", port: 6660 }
2024-09-05T09:22:04.891390513Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 3
2024-09-05T09:22:04.891442351Z  INFO risingwave_meta::hummock::manager::compaction: compactor 3 enters the cluster

### compute-0

2024-09-05T09:19:58.065542097Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097076570193920 curr_epoch=70970766357299
20}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multi
ple barriers at once in now executor: 2
2024-09-05T09:20:25.836565253Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:20:28.586230387Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.593286697Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.664237551Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.664738403Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.679278103Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.716121816Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.869541767Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:28.933198464Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097078523232256 curr_epoch=7097078588768256}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:20:55.835271236Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:20:57.809021751Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:20:57.835563982Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multipl:
e barriers at once in now executor: 2
2024-09-05T09:20:57.89438558Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:20:57.897841327Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:20:57.897973129Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:20:57.917684026Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:20:58.087889308Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:20:58.345024603Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097080491147264 curr_epoch=7097080556748800}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:21:25.838192587Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:21:27.658544401Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:27.755803377Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:27.790127437Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:27.838639786Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:27.868806303Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:27.883569938Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:27.9687079Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:21:28.349264283Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097082466795520 curr_epoch=7097082532331520}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multipl
e barriers at once in now executor: 3
2024-09-05T09:21:56.83550108Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:25.835214895Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:28.8368923Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:29.177538886Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:22:29.231436239Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:22:29.262094543Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:22:29.305917283Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:22:29.355732618Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:22:29.361047429Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:22:29.395234759Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:22:29.964588119Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097086398955520 curr_epoch=7097086464491520}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:22:55.835522967Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:57.362513998Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:22:57.431440508Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:22:57.504583308Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:22:57.519820494Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:22:57.52350527Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multip
le barriers at once in now executor: 2
2024-09-05T09:22:57.561583259Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:22:57.70943515Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:22:58.049160192Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097088354746368 curr_epoch=7097088420282368}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:23:25.835447903Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:23:28.837039495Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:23:29.081467516Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:23:29.191689551Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:23:29.204486979Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:23:29.312832821Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:23:29.432629192Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:23:29.432822673Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:23:29.443211009Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:23:29.706918718Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097090321219584 curr_epoch=7097090386821120}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:23:55.836095109Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:23:58.837222543Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:24:00.592364733Z  WARN actor{otel.name="Actor 497" actor_id=497 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project 1F100000002"}:executor{otel.name="Now 1F100002738"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:00.650043819Z  WARN actor{otel.name="Actor 187" actor_id=187 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project BB00000003"}:executor{otel.name="Now BB00002756"}: risingwave_stream::executor::now: handle multipl:
e barriers at once in now executor: 5
2024-09-05T09:24:00.722229203Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:00.743880345Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:00.744022529Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:00.76653565Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:00.889663696Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:01.099061611Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097092290772992 curr_epoch=7097092356243456}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 5
2024-09-05T09:24:29.825672315Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097094387924992 curr_epoch=7097094453460992}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:24:29.836298895Z  WARN actor{otel.name="Actor 153" actor_id=153 prev_epoch=7097094387924992 curr_epoch=7097094453460992}:executor{otel.name="Project 9900000003"}:executor{otel.name="Now 9900002748"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:24:29.8513241Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097094322388992 curr_epoch=7097094387924992}:executor{otel.name="Project 1F300000004"}:executor{otel.name="Now 1F30000276F"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:24:55.836536078Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:25:25.835817269Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:25:27.159412607Z  WARN actor{otel.name="Actor 228" actor_id=228 prev_epoch=7097098183639040 curr_epoch=7097098249175040}:executor{otel.name="Project E400000002"}:executor{otel.name="Now E4000027A4"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:25:27.159579201Z  WARN actor{otel.name="Actor 294" actor_id=294 prev_epoch=7097098183639040 curr_epoch=7097098249175040}:executor{otel.name="Project 12600000003"}:executor{otel.name="Now 1260000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:25:27.162150679Z  WARN actor{otel.name="Actor 311" actor_id=311 prev_epoch=7097098183639040 curr_epoch=7097098249175040}:executor{otel.name="Project 13700000003"}:executor{otel.name="Now 13700002764"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:25:27.179129199Z  WARN actor{otel.name="Actor 498" actor_id=498 prev_epoch=7097098183639040 curr_epoch=7097098249175040}:executor{otel.name="Project 1F200000003"}:executor{otel.name="Now 1F20000276A"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:25:27.242109625Z  WARN actor{otel.name="Actor 499" actor_id=499 prev_epoch=7097098183639040 curr_epoch=70970982491750:

### compute-1

2024-09-05T09:20:25.850551563Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:20:28.543340608Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097078457630720 curr_epoch=7097078523232256}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:20:55.852306357Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:20:57.747604331Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097080425676800 curr_epoch=7097080491147264}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:21:25.850375464Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:21:27.556250566Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097082401128448 curr_epoch=7097082466795520}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:21:56.852278295Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:25.850930439Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:28.852260777Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:29.122207002Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097086333419520 curr_epoch=7097086398955520}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:22:56.850222538Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:22:57.334003885Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097088289144832 curr_epoch=7097088354746368}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3
2024-09-05T09:23:26.850230654Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:23:29.023762038Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097090255749120 curr_epoch=7097090321219584}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 4
2024-09-05T09:23:30.345689984Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097090517893120 curr_epoch=7097090583429120}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:23:55.850948009Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:23:58.852213799Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:24:00.865229375Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097092225236992 curr_epoch=7097092290772992}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 6
2024-09-05T09:24:29.782677396Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097094387924992 curr_epoch=7097094453460992}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:24:56.426390993Z  WARN actor{otel.name="Actor 1352" actor_id=1352 prev_epoch=7097096170242048 curr_epoch=7097096235843584}:executor{otel.name="Project 54800000005"}:executor{otel.name="Now 54800002759"}: risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-09-05T09:25:26.850679189Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:25:43.947255255Z  INFO risingwave_common::telemetry::report: Telemetry tracking_id is not set, event reporting disabled
2024-09-05T09:25:43.950749565Z  INFO risingwave_common::telemetry::report: Telemetry tracking_id is not set, event reporting disabled
2024-09-05T09:25:43.951343496Z  INFO risingwave_common::telemetry::report: Telemetry tracking_id is not set, event reporting disabled
2024-09-05T09:25:43.951931022Z  INFO risingwave_common::telemetry::report: Telemetry tracking_id is not set, event reporting disabled
2024-09-05T09:25:44.347378818Z  INFO actor{otel.name="Actor 1372" actor_id=1372}:executor{otel.name="Project 55C00000004"}:executor{otel.name="RowIdGen 55C00000003"}:executor{otel.name="WatermarkFilter 55C00000002"}:executor{otel.name="Project 55C00000001"}:executor{otel.name="Source 55C00002731"}:build:new:new:connect:connect_inner: pulsar::connection_manager: Connected n°06ab1c0c-40fb-497c-bb91-6062b2d2fd46 to pulsar://pulsarhost:6850 in 331ms
2024-09-05T09:25:45.01803133Z  INFO actor{otel.name="Actor 1372" actor_id=1372}:executor{otel.name="Project 55C00000004"}:executor{otel.name="RowIdGen 55C00000003"}:executor{otel.name="WatermarkFilter 55C00000002"}:executor{otel.name="Project 55C00000001"}:executor{otel.name="Source 55C00002731"}:build:validate:lookup_partitioned_topic:lookup_partitioned_topic:lookup_topic:get_connection:connect:connect_inner: pulsar::connection_manager: Connected n°cba4d774-fefc-402e-81ce-a9b8d5cfea72 to pulsarhost:6650 via proxy pulsar://pulsarhost:6850 in 333ms
2024-09-05T09:25:45.01821452Z  WARN actor{otel.name="Actor 1372" actor_id=1372}:executor{otel.name="Project 55C00000004"}:executor{otel.name="RowIdGen 55C00000003"}:executor{otel.name="WatermarkFilter 55C00000002"}:executor{otel.name="Project 55C00000001"}:executor{otel.name="Source 55C00002731"}:build:new: pulsar::retry_op: Retry #0 -> connecting consumer 4 using connection cba4d774-fefc-402e-81ce-a9b8d5cfea72 to broker pulsar://pulsarhost:6850 to topic persistent://tenant/mytopic
2024-09-05T09:25:59.233038802Z  WARN pulsar::consumer::engine: rx terminated
2024-09-05T09:25:59.232832394Z ERROR engine:process_message:process_payload:send_to_consumer: pulsar::consumer::engine: tx returned SendError { kind: Disconnected }
2024-09-05T09:25:59.235024453Z ERROR engine: pulsar::consumer::engine: cannot send a message from the consumer engine to the consumer(4), stopping the engine
2024-09-05T09:25:59.235260271Z ERROR pulsar::consumer::engine: Error sending end event to channel - send failed because receiver is gone
2024-09-05T09:25:59.235273857Z  WARN pulsar::consumer::engine: rx terminated
2024-09-05T09:26:26.85074319Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:26:55.850336336Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:28:25.850375012Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:29:25.85022756Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:29:55.851368419Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:29:59.978908147Z  INFO lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took Ok(17.358µs))
2024-09-05T09:37:39.346313035Z  INFO risingwave_stream::task::stream_manager: start inspecting barrier state
2024-09-05T09:37:39.383990602Z  INFO risingwave_stream::task::stream_manager: finish inspecting barrier state ok=true time=37.621791ms
2024-09-05T09:38:24.697668562Z  WARN risingwave_common_service::observer_manager: Receives meta's notification err error=status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }: error reading a body from connection: stream closed because of a broken pipe
2024-09-05T09:38:24.697849726Z  WARN risingwave_common_service::observer_manager: Receives meta's notification err error=status: Unk

### compactor-1

2024-09-05T09:19:58.747592039Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pendin
g_pull_task_count=0
2024-09-05T09:19:59.748138839Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pendin
g_pull_task_count=4
2024-09-05T09:19:59.814453368Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485678 compact
_task_statistics CompactTaskStatistics { total_file_count: 27, total_key_count: 19738, total_file_size: 1410657, total_uncompressed
_file_size: 1396878 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 60586360, enable fast runner: fa
lse, Compaction task id: 485678, group-id: 2, type: Dynamic, target level: 0, target sub level: 7097076042235904 watermark: 7097076
304379904, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 843504, obj_id: 843504 32KB stale_ratio 0]", "[id: 843495, obj_id: 843495 34KB stale_ratio 0]"]
Level 0 ["[id: 843494, obj_id: 843494 139KB stale_ratio 0]", "[id: 843503, obj_id: 843503 117KB stale_ratio 0]"]
Level 0 ["[id: 843501, obj_id: 843501 36KB stale_ratio 0]", "[id: 843492, obj_id: 843492 35KB stale_ratio 0]"]
Level 0 ["[id: 843500, obj_id: 843500 32KB stale_ratio 0]", "[id: 843491, obj_id: 843491 34KB stale_ratio 0]"]
Level 0 ["[id: 843498, obj_id: 843498 36KB stale_ratio 0]", "[id: 843490, obj_id: 843490 37KB stale_ratio 0]"]
Level 0 ["[id: 843499, obj_id: 843499 32KB stale_ratio 0]", "[id: 843487, obj_id: 843487 34KB stale_ratio 0]"]
Level 0 ["[id: 843497, obj_id: 843497 33KB stale_ratio 0]", "[id: 843489, obj_id: 843489 102KB stale_ratio 0]"]
Level 0 ["[id: 843484, obj_id: 843484 36KB stale_ratio 0]", "[id: 843475, obj_id: 843475 37KB stale_ratio 0]"]
Level 0 ["[id: 843474, obj_id: 843474 49KB stale_ratio 0]", "[id: 843482, obj_id: 843482 74KB stale_ratio 0]", "[id: 843483, obj_id
: 843483 3KB stale_ratio 0]"]
Level 0 ["[id: 843480, obj_id: 843480 40KB stale_ratio 0]", "[id: 843472, obj_id: 843472 44KB stale_ratio 0]"]
Level 0 ["[id: 843478, obj_id: 843478 32KB stale_ratio 0]", "[id: 843470, obj_id: 843470 34KB stale_ratio 0]"]
Level 0 ["[id: 843477, obj_id: 843477 33KB stale_ratio 0]", "[id: 843469, obj_id: 843469 175KB stale_ratio 0]"]
Level 0 ["[id: 843455, obj_id: 843455 36KB stale_ratio 0]", "[id: 843467, obj_id: 843467 37KB stale_ratio 0]"]

2024-09-05T09:19:59.825245457Z  INFO lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occ
urred; added new identity (took Ok(34.503µs))
2024-09-05T09:20:00.231965236Z  INFO risingwave_storage::hummock::compactor: Finish Task Some(485678) split_index Some(0) sst count 1
2024-09-05T09:20:00.232224106Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Finished compaction task in 418.10197300000004ms: Compaction task id: 485678, group-id: 2, type: Dynamic, target level: 0, target sub level: 7097076042235904 watermark: 7097076304379904, target_file_size: 33554432, splits: 1, status: Success
Output:
SstableInfo: object id=843507, SST id=843507, KeyRange=["00000011000000017065727369737465096e743a2f2f707269096d652d7472616469096e672f7072696d65092d706f736974696f096e2d646174612f7309796e746865746963092d6f65732d62792d096f69642d6a736f6e08001936c0f3a10000","000002e90000001936c0d8110000"], table_ids: [17, 23, 24, 26, 28, 30, 37, 39, 44, 45, 53, 55, 60, 62, 65, 66, 73, 76, 79, 80, 85, 87, 94, 95, 96, 97, 100, 101, 102, 103, 104, 108, 109, 110, 111, 112, 114, 115, 116, 117, 119, 123, 124, 125, 126, 128, 130, 136, 139, 247, 279, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680
, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 740, 741, 742, 744, 745], size=1162KB, stale_ratio=5%, bloom_filter_kind 1

2024-09-05T09:20:00.747603905Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:00.896971362Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485681 compact_task_statistics CompactTaskStatistics { total_file_count: 3, total_key_count: 58167, total_file_size: 3546677, total_uncompressed_file_size: 3539108 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 10419704, enable fast runner: false, Compaction task id: 485681, group-id: 2, type: Dynamic, target level: 0, target sub level: 7097074408947712 watermark: 7097076304379904, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 843507, obj_id: 843507 1162KB stale_ratio 5]"]
Level 0 ["[id: 843457, obj_id: 843457 896KB stale_ratio 2]"]
Level 0 ["[id: 843368, obj_id: 843368 1404KB stale_ratio 0]"]

2024-09-05T09:20:01.448203158Z  INFO risingwave_storage::hummock::compactor: Finish Task Some(485681) split_index Some(0) sst count 1
2024-09-05T09:20:01.448499026Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Finished compaction task in 551.7423070000001ms: Compaction task id: 485681, group-id: 2, type: Dynamic, target level: 0, target sub level: 7097074408947712 watermark: 7097076304379904, target_file_size: 33554432, splits: 1, status: Success
Output:
SstableInfo: object id=843508, SST id=843508, KeyRange=["00000011000000017065727369737465096e743a2f2f707269096d652d7472616469096e672f7072696d65092d706f736974696f096e2d646174612f7309796e746865746963092d6f65732d62792d096f69642d6a736f6e08001936c0f3a10000","000002e90000001936c0d8110000"], table_ids: [17, 23, 24, 26, 28, 30, 37, 39, 44, 45, 53, 55, 60, 62, 65, 66, 73, 76, 79, 80, 85, 87, 94, 95, 96, 97, 100, 101, 102, 103, 104, 108, 109, 110, 111, 112, 114, 115, 116, 117, 119, 123, 124, 125, 126, 128, 130, 136, 139, 247, 279, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 740, 741, 742, 744, 745], size=1598KB, stale_ratio=3%, bloom_filter_kind 1

2024-09-05T09:20:01.748447663Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:01.834444436Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485684 compact_task_statistics CompactTaskStatistics { total_file_count: 12, total_key_count: 90472, total_file_size: 105198074, total_uncompressed_file_size: 105183170 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 53808361, enable fast runner: false, Compaction task id: 485684, group-id: 3, type: Dynamic, target level: 0, target sub level: 7096082762432512 watermark: 7097076304379904, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 843369, obj_id: 843369 63KB stale_ratio 28]"]
Level 0 ["[id: 843338, obj_id: 843338 62KB stale_ratio 0]"]
Level 0 ["[id: 843188, obj_id: 843188 57KB stale_ratio 5]"]
Level 0 ["[id: 843048, obj_id: 843048 76KB stale_ratio 35]"]
Level 0 ["[id: 842977, obj_id: 842977 49KB stale_ratio 26]"]
Level 0 ["[id: 842810, obj_id: 842810 36KB stale_ratio 5]"]
Level 0 ["[id: 842671, obj_id: 842671 65KB stale_ratio 14]"]
Level 0 ["[id: 842670, obj_id: 842670 1242KB stale_ratio 0]"]
Level 0 ["[id: 842365, obj_id: 842365 15966KB stale_ratio 0]", "[id: 842366, obj_id: 842366 32761KB stale_ratio 0]", "[id: 842667, obj_id: 842667 24143KB stale_ratio 0]", "[id: 842668, obj_id: 842668 28207KB stale_ratio 0]"]
Table vnode partition info:
 [631, 1]

2024-09-05T09:20:01.834712963Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485683 compact_task_statistics CompactTaskStatistics { total_file_count: 12, total_key_count: 376635, total_file_size: 30805655, total_uncompressed_file_size: 30774488 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 59324052, enable fast runner: false, Compaction task id: 485683, group-id: 2, type: Dynamic, target level: 0, target sub level: 7096293653938176 watermark: 7097076304379904, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 843508, obj_id: 843508 1598KB stale_ratio 3]"]
Level 0 ["[id: 843367, obj_id: 843367 941KB stale_ratio 5]"]
Level 0 ["[id: 843337, obj_id: 843337 1079KB stale_ratio 4]"]
Level 0 ["[id: 843297, obj_id: 843297 1165KB stale_ratio 2]"]
Level 0 ["[id: 843189, obj_id: 843189 1100KB stale_ratio 5]"]
Level 0 ["[id: 843187, obj_id: 843187 1194KB stale_ratio 2]"]
Level 0 ["[id: 843051, obj_id: 843051 1409KB stale_ratio 6]"]
Level 0 ["[id: 842948, obj_id: 842948 1166KB stale_ratio 2]"]
Level 0 ["[id: 842947, obj_id: 842947 1064KB stale_ratio 2]"]
Level 0 ["[id: 842897, obj_id: 842897 1025KB stale_ratio 7]"]
Level 0 ["[id: 842809, obj_id: 842809 1155KB stale_ratio 4]"]
Level 0 ["[id: 842808, obj_id: 842808 17182KB stale_ratio 13]"]

2024-09-05T09:20:01.886875493Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485686 compact_task_statistics CompactTaskStatistics { total_file_count: 101, total_key_count: 235003, total_file_size: 170851631, total_uncompressed_file_size: 170787542 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 300400790, enable fast runner: false, Compaction task id: 485686, group-id: 2, type: Dynamic, target level: 0, target sub level: 7095766169550848 watermark: 7097076304379904, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 776070, obj_id: 776070 32KB stale_ratio 0]", "[id: 776078, obj_id: 776078 34KB stale_ratio 0]"]
Level 0 ["[id: 776069, obj_id: 776069 1767KB stale_ratio 0]", "[id: 776077, obj_id: 776077 3171KB stale_ratio 0]"]
Level 0 ["[id: 776067, obj_id: 776067 64KB stale_ratio 0]", "[id: 776065, obj_id: 776065 121KB stale_ratio 0]"]
Level 0 ["[id: 776055, obj_id: 776055 49KB stale_ratio 0]", "[id: 776063, obj_id: 776063 4311KB stale_ratio 0]"]
Level 0 ["[id: 776008, obj_id: 776008 21848KB stale_ratio 3]"]
Level 0 ["[id: 776007, obj_id: 776007 17756KB stale_ratio 4]"]
Level 0 ["[id: 775926, obj_id: 775926 32KB stale_ratio 0]", "[id: 775934, obj_id: 775934 34KB stale_ratio 0]"]
Level 0 ["[id: 775924, obj_id: 775924 55KB stale_ratio 0]", "[id: 775932, obj_id: 775932 88KB stale_ratio 0]"]
Level 0 ["[id: 775922, obj_id: 775922 57KB stale_ratio 0]", "[id: 775931, obj_id: 775931 47KB stale_ratio 0]"]
Level 0 ["[id: 775929, obj_id: 775929 1986KB stale_ratio 0]", "[id: 775921, obj_id: 775921 3006KB stale_ratio 0]"]
Level 0 ["[id: 775919, obj_id: 775919 32KB stale_ratio 0]", "[id: 775927, obj_id: 775927 34KB stale_ratio 0]"]
Level 0 ["[id: 775916, obj_id: 775916 2622KB stale_ratio 0]", "[id: 775918, obj_id: 775918 1780KB stale_ratio 0]"]
Level 0 ["[id: 775914, obj_id: 775914 3653KB stale_ratio 0]", "[id: 775905, obj_id: 775905 1275KB stale_ratio 0]"]
Level 0 ["[id: 775869, obj_id: 775869 15986KB stale_ratio 5]"]
Level 0 ["[id: 775868, obj_id: 775868 5301KB stale_ratio 3]"]
Level 0 ["[id: 775796, obj_id: 775796 1313KB stale_ratio 0]", "[id: 775803, obj_id: 775803 3081KB stale_ratio 0]"]
Level 0 ["[id: 775794, obj_id: 775794 83KB stale_ratio 0]", "[id: 775800, obj_id: 775800 67KB stale_ratio 0]"]
Level 0 ["[id: 775799, obj_id: 775799 3302KB stale_ratio 0]", "[id: 775792, obj_id: 775792 1168KB stale_ratio 0]"]
Level 0 ["[id: 775797, obj_id: 775797 34KB stale_ratio 0]", "[id: 775790, obj_id: 775790 32KB stale_ratio 0]"]
Level 0 ["[id: 775786, obj_id: 775786 34KB stale_ratio 0]", "[id: 775789, obj_id: 775789 32KB stale_ratio 0]"]
Level 0 ["[id: 775788, obj_id: 775788 1401KB stale_ratio 0]", "[id: 775785, obj_id: 775785 3711KB stale_ratio 0]"]
Level 0 ["[id: 775783, obj_id: 775783 34KB stale_ratio 0]", "[id: 775776, obj_id: 775776 36KB stale_ratio 0]"]
Level 0 ["[id: 775782, obj_id: 775782 2672KB stale_ratio 0]", "[id: 775775, obj_id: 775775 1815KB stale_ratio 0]"]
Level 0 ["[id: 775773, obj_id: 775773 32KB stale_ratio 0]", "[id: 775780, obj_id: 775780 34KB stale_ratio 0]"]
Level 0 ["[id: 775772, obj_id: 775772 32KB stale_ratio 0]", "[id: 775779, obj_id: 775779 34KB stale_ratio 0]"]
Level 0 ["[id: 775771, obj_id: 775771 32KB stale_ratio 0]", "[id: 775778, obj_id: 775778 34KB stale_ratio 0]"]
Level 0 ["[id: 775769, obj_id: 775769 44KB stale_ratio 0]", "[id: 775777, obj_id: 775777 54KB stale_ratio 0]"]
Level 0 ["[id: 775603, obj_id: 775603 14682KB stale_ratio 4]"]
Level 0 ["[id: 775600, obj_id: 775600 19691KB stale_ratio 3]"]
Level 0 ["[id: 775655, obj_id: 775655 32KB stale_ratio 0]", "[id: 775662, obj_id: 775662 32KB stale_ratio 0]"]
Level 0 ["[id: 775654, obj_id: 775654 32KB stale_ratio 0]", "[id: 775661, obj_id: 775661 32KB stale_ratio 0]"]
Level 0 ["[id: 775653, obj_id: 775653 32KB stale_ratio 0]", "[id: 775660, obj_id: 775660 32KB stale_ratio 0]"]
Level 0 ["[id: 775652, obj_id: 775652 47KB stale_ratio 0]", "[id: 775659, obj_id: 775659 39KB stale_ratio 0]"]
Level 0 ["[id: 775650, obj_id: 775650 32KB stale_ratio 0]", "[id: 775657, obj_id: 775657 34KB stale_ratio 0]"]
Level 0 ["[id: 775646, obj_id: 775646 34KB stale_ratio 0]", "[id: 775649, obj_id: 775649 32KB stale_ratio 0]"]
Level 0 ["[id: 775645, obj_id: 775645 34KB stale_ratio 0]", "[id: 775648, obj_id: 775648 32KB stale_ratio 0]"]
Level 0 ["[id: 775643, obj_id: 775643 987KB stale_ratio 0]", "[id: 775636, obj_id: 775636 60KB stale_ratio 0]"]
Level 0 ["[id: 775634, obj_id: 775634 33KB stale_ratio 0]", "[id: 775640, obj_id: 775640 71KB stale_ratio 0]", "[id: 775641, obj_id: 775641 0KB stale_ratio 0]"]
Level 0 ["[id: 775633, obj_id: 775633 1439KB stale_ratio 0]", "[id: 775639, obj_id: 775639 3727KB stale_ratio 1]"]
Level 0 ["[id: 775631, obj_id: 775631 82KB stale_ratio 0]", "[id: 775637, obj_id: 775637 99KB stale_ratio 0]"]
Level 0 ["[id: 775630, obj_id: 775630 32KB stale_ratio 0]", "[id: 775626, obj_id: 775626 34KB stale_ratio 0]"]
Level 0 ["[id: 775629, obj_id: 775629 32KB stale_ratio 0]", "[id: 775625, obj_id: 775625 34KB stale_ratio 0]"]
Level 0 ["[id: 775624, obj_id: 775624 37KB stale_ratio 0]", "[id: 775627, obj_id: 775627 32KB stale_ratio 0]"]
Level 0 ["[id: 775623, obj_id: 775623 34KB stale_ratio 0]", "[id: 775628, obj_id: 775628 32KB stale_ratio 0]"]
Level 0 ["[id: 775596, obj_id: 775596 32KB stale_ratio 0]", "[id: 775622, obj_id: 775622 34KB stale_ratio 0]"]
Level 0 ["[id: 775594, obj_id: 775594 1876KB stale_ratio 0]", "[id: 775620, obj_id: 775620 3222KB stale_ratio 1]"]
Level 0 ["[id: 775592, obj_id: 775592 2663KB stale_ratio 0]", "[id: 775619, obj_id: 775619 4001KB stale_ratio 0]"]
Level 0 ["[id: 775591, obj_id: 775591 1432KB stale_ratio 0]", "[id: 775617, obj_id: 775617 2858KB stale_ratio 1]"]
Level 0 ["[id: 775499, obj_id: 775499 129KB stale_ratio 0]", "[id: 775494, obj_id: 775494 82KB stale_ratio 0]"]
Level 0 ["[id: 775493, obj_id: 775493 37KB stale_ratio 0]", "[id: 775497, obj_id: 775497 36KB stale_ratio 0]"]
Level 0 ["[id: 775476, obj_id: 775476 32KB stale_ratio 0]", "[id: 775492, obj_id: 775492 34KB stale_ratio 0]"]
Level 0 ["[id: 775475, obj_id: 775475 32KB stale_ratio 0]", "[id: 775491, obj_id: 775491 34KB stale_ratio 0]"]
Level 0 ["[id: 775474, obj_id: 775474 99KB stale_ratio 0]", "[id: 775490, obj_id: 775490 4299KB stale_ratio 0]"]
Dropped table_ids: {628, 625, 627, 626, 629}

2024-09-05T09:20:02.347307928Z  WARN risingwave_storage::hummock::sstable::builder: A large key/value (table_id=247, key len=902351, value len=0, epoch=7096299887525888, spill offset=0) is added to block
2024-09-05T09:20:02.364295957Z  WARN risingwave_storage::hummock::sstable::builder: A large key/value (table_id=247, key len=885314, value len=0, epoch=7096299624202240, spill offset=0) is added to block
2024-09-05T09:20:02.388555495Z  WARN risingwave_storage::hummock::sstable::builder: A large key/value (table_id=247, key len=925310, value len=696065, epoch=7096299887525888, spill offset=0) is added to block
thread 'rw-compaction' panicked at /risingwave/src/storage/hummock_sdk/src/key.rs:1089:21:
key UserKey { 10, TableKey { 000000017065727369737465096e743a2f2f707269096d652d7472616469096e672d71612f707509626c69632d61706109632d636f6d6d616e09642f66786d5f6e7509766f5f726174655f0961737369676e6d65096e742d6a736f6e0007 } } epoch EpochWithGap(7095770881916928) >= prev epoch EpochWithGap(7095770881916928)
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:72:14
   2: risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe_multi_version
             at ./risingwave/src/storage/hummock_sdk/src/key.rs:1089:21
   3: risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe
             at ./risingwave/src/storage/hummock_sdk/src/key.rs:1038:9
   4: risingwave_storage::hummock::compactor::compactor_runner::compact_and_build_sst::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:662:31
   5: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
   6: risingwave_storage::hummock::compactor::Compactor::compact_key_range_impl::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/mod.rs:262:10
   7: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
   8: risingwave_storage::hummock::compactor::Compactor::compact_key_range::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/mod.rs:171:18
   9: risingwave_storage::hummock::compactor::compactor_runner::CompactorRunner::run::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:152:14
  10: risingwave_storage::hummock::compactor::compactor_runner::compact::{{closure}}::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:489:18
  11: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  12: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  13: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  14: await_tree::root::TreeRoot::instrument::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/root.rs:43:34
  15: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
  16: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  22: std::panicking::try::do_call
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  23: std::panicking::try
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  24: std::panic::catch_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  25: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  26: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  27: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  28: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  29: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  30: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:639:22
  31: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  32: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  33: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  36: tokio::runtime::context::scoped::Scoped<T>::set
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  37: tokio::runtime::context::set_scheduler::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  38: std::thread::local::LocalKey<T>::try_with
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:284:16
  39: std::thread::local::LocalKey<T>::with
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:260:9
  40: tokio::runtime::context::set_scheduler
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17
  41: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  42: tokio::runtime::context::runtime::enter_runtime
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  43: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  44: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  45: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  46: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  47: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  48: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  49: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  50: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19

  51: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  52: std::panicking::try::do_call
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  53: std::panicking::try
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  54: std::panic::catch_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  55: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  56: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  57: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  58: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  59: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  60: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  61: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  62: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
2024-09-05T09:20:02.748198014Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=3 pull_task_ack=false pending_pull_task_count=4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

*** await tree context of current task ***

Compaction Task 485686 Split 0  [890.084ms]
  compact [887.084ms]
    compact_and_build_sst [887.084ms]  <== current

opentelemetry tracing will be exported to `http://host1:5690/` if enabled
2024-09-05T09:20:03.142730466Z  INFO risingwave_rt::panic_hook: coredump on panic is likely to be enabled limit=18446744073709551615

#### compactor-0

2024-09-05T09:19:55.89724957Z  WARN risingwave_rpc_client::meta_client: Failed to send_heartbeat: timeout
2024-09-05T09:19:56.896523552Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pendin
g_pull_task_count=0
2024-09-05T09:19:57.896881696Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pendin
g_pull_task_count=4
2024-09-05T09:19:58.895952907Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pendin
g_pull_task_count=0
2024-09-05T09:19:59.896016298Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pendin
g_pull_task_count=4
2024-09-05T09:20:00.896615421Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:01.89659309Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:02.89631957Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:03.896414618Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:04.896522202Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:05.896045612Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:06.896491821Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:07.896445935Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:07.932647642Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485696 compact_task_statistics CompactTaskStatistics { total_file_count: 24, total_key_count: 69, total_file_size: 74818, total_uncompressed_file_size: 74332 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 52001955, enable fast runner: false, Compaction task id: 485696, group-id: 3, type: Dynamic, target level: 0, target sub level: 7097075976699904 watermark: 7097077100380160, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 843536, obj_id: 843536 1KB stale_ratio 0]", "[id: 843538, obj_id: 843538 3KB stale_ratio 0]"]
Level 0 ["[id: 843533, obj_id: 843533 3KB stale_ratio 0]", "[id: 843524, obj_id: 843524 3KB stale_ratio 0]"]
Level 0 ["[id: 843530, obj_id: 843530 1KB stale_ratio 0]", "[id: 843521, obj_id: 843521 3KB stale_ratio 0]"]
Level 0 ["[id: 843519, obj_id: 843519 1KB stale_ratio 0]", "[id: 843528, obj_id: 843528 2KB stale_ratio 0]"]
Level 0 ["[id: 843517, obj_id: 843517 3KB stale_ratio 0]", "[id: 843506, obj_id: 843506 1KB stale_ratio 0]"]
Level 0 ["[id: 843493, obj_id: 843493 3KB stale_ratio 0]", "[id: 843502, obj_id: 843502 7KB stale_ratio 0]"]
Level 0 ["[id: 843486, obj_id: 843486 1KB stale_ratio 0]", "[id: 843488, obj_id: 843488 3KB stale_ratio 0]"]
Level 0 ["[id: 843485, obj_id: 843485 1KB stale_ratio 0]", "[id: 843476, obj_id: 843476 2KB stale_ratio 0]"]
Level 0 ["[id: 843473, obj_id: 843473 3KB stale_ratio 0]", "[id: 843481, obj_id: 843481 1KB stale_ratio 0]"]
Level 0 ["[id: 843479, obj_id: 843479 2KB stale_ratio 0]", "[id: 843471, obj_id: 843471 4KB stale_ratio 0]"]
Level 0 ["[id: 843456, obj_id: 843456 1KB stale_ratio 0]", "[id: 843468, obj_id: 843468 2KB stale_ratio 0]"]
Level 0 ["[id: 843444, obj_id: 843444 3KB stale_ratio 0]", "[id: 843452, obj_id: 843452 4KB stale_ratio 0]"]

2024-09-05T09:20:08.047753542Z  INFO risingwave_storage::hummock::compactor: Finish Task Some(485696) split_index Some(0) sst count 1
2024-09-05T09:20:08.047877122Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Finished compaction task in 115.30244ms: Compaction task id: 485696, group-id: 3, type: Dynamic, target level: 0, target sub level: 7097075976699904 watermark: 7097077100380160, target_file_size: 33554432, splits: 1, status: Success
Output:
SstableInfo: object id=843458, SST id=843458, KeyRange=["0000000f00730001333234303930343009303133333339333908001936c0f3a10000","000002e300dc0001333234303930343009303032383837333008001936c10f570000"], table_ids: [15, 22, 70, 86, 105, 515, 631, 739], size=41KB, stale_ratio=17%, bloom_filter_kind 1

2024-09-05T09:20:08.895798356Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:09.896098256Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:10.89873622Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:11.895805273Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:11.936753755Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485705 compact_task_statistics CompactTaskStatistics { total_file_count: 25, total_key_count: 18047, total_file_size: 1272849, total_uncompressed_file_size: 1261023 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 56041266, enable fast runner: false, Compaction task id: 485705, group-id: 2, type: Dynamic, target level: 0, target sub level: 7097076902461440 watermark: 7097077100380160, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 843554, obj_id: 843554 33KB stale_ratio 0]", "[id: 843543, obj_id: 843543 110KB stale_ratio 0]"]
Level 0 ["[id: 843541, obj_id: 843541 44KB stale_ratio 0]", "[id: 843551, obj_id: 843551 44KB stale_ratio 0]"]
Level 0 ["[id: 843550, obj_id: 843550 32KB stale_ratio 0]", "[id: 843540, obj_id: 843540 34KB stale_ratio 0]"]
Level 0 ["[id: 843549, obj_id: 843549 32KB stale_ratio 0]", "[id: 843539, obj_id: 843539 34KB stale_ratio 0]"]
Level 0 ["[id: 843547, obj_id: 843547 78KB stale_ratio 0]", "[id: 843548, obj_id: 843548 3KB stale_ratio 0]", "[id: 843537, obj_id: 843537 52KB stale_ratio 0]"]
Level 0 ["[id: 843526, obj_id: 843526 34KB stale_ratio 0]", "[id: 843535, obj_id: 843535 32KB stale_ratio 0]"]
Level 0 ["[id: 843534, obj_id: 843534 40KB stale_ratio 0]", "[id: 843525, obj_id: 843525 44KB stale_ratio 0]"]
Level 0 ["[id: 843532, obj_id: 843532 32KB stale_ratio 0]", "[id: 843523, obj_id: 843523 34KB stale_ratio 0]"]
Level 0 ["[id: 843531, obj_id: 843531 86KB stale_ratio 0]", "[id: 843522, obj_id: 843522 50KB stale_ratio 0]"]
Level 0 ["[id: 843520, obj_id: 843520 37KB stale_ratio 0]", "[id: 843529, obj_id: 843529 36KB stale_ratio 0]"]
Level 0 ["[id: 843518, obj_id: 843518 50KB stale_ratio 0]", "[id: 843527, obj_id: 843527 187KB stale_ratio 0]"]
Level 0 ["[id: 843505, obj_id: 843505 36KB stale_ratio 0]", "[id: 843496, obj_id: 843496 37KB stale_ratio 0]"]

2024-09-05T09:20:12.406537502Z  INFO risingwave_storage::hummock::compactor: Finish Task Some(485705) split_index Some(0) sst count 1
2024-09-05T09:20:12.407117239Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Finished compaction task in 470.78130899999996ms: Compaction task id: 485705, group-id: 2, type: Dynamic, target level: 0, target sub level: 7097076902461440 watermark: 7097077100380160, target_file_size: 33554432, splits: 1, status: Success
Output:
SstableInfo: object id=843459, SST id=843459, KeyRange=["00000011000000017065727369737465096e743a2f2f707269096d652d7472616469096e672f7072696d65092d706f736974696f096e2d646174612f7309796e746865746963092d6f65732d62792d096f69642d6a736f6e08001936c126cb0000","000002e90000001936c107830000"], table_ids: [17, 23, 24, 26, 28, 30, 37, 39, 44, 45, 53, 55, 60, 62, 65, 66, 73, 76, 79, 80, 85, 87, 94, 95, 96, 97, 100, 101, 102, 103, 104, 108, 109, 110, 111, 112, 114, 115, 116, 117, 119, 123, 124, 125, 126, 128, 130, 136, 139, 247, 279, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 740, 741, 742, 744, 745], size=867KB, stale_ratio=6%, bloom_filter_kind 1

2024-09-05T09:20:12.900405401Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:13.895929771Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:14.896085898Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:15.897728921Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:16.895865Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:17.896105032Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:18.443776904Z  INFO risingwave_storage::hummock::vacuum: Try to vacuum SSTs [843205, 843229, 843313, 843225, 843316, 843181, 843431, 843440, 843396, 843216, 843333, 843385, 843286, 843358, 843329, 843443, 843407, 843294, 843401, 843424, 843212, 843429, 843296, 843448, 843406, 843450, 843240, 843199, 843173, 843349, 843377, 843366, 843437, 843282, 843211, 843414, 843426, 843418, 843428, 843253, 843394, 843290, 843415, 843398, 843423, 843432, 843445, 843389, 843420, 843451, 843436, 843446, 843411, 843392, 843364, 843433, 843404, 843250, 843422, 843236, 843197, 843379, 843384, 843427, 843218, 843387, 843267, 843409, 843311, 843202, 843421, 843449, 843395, 843399, 843287, 843408, 843231, 843453, 843207, 843402, 843204, 843378, 843221, 843331, 843442, 843400, 843383, 843425, 843322, 843434, 843447, 843185, 843354, 843438, 843391, 843397, 843386, 843403, 843413, 843208, 843417, 843454, 843405, 843244, 843435, 843382, 843430, 843380, 843390, 843381, 843441, 843388, 843416, 843412, 843410, 843419, 843439, 843319, 843393]
2024-09-05T09:20:18.844896993Z  INFO risingwave_storage::hummock::vacuum: Finished vacuuming SSTs
2024-09-05T09:20:18.89609354Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:19.895950327Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4

2024-09-05T09:20:20.896112934Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-09-05T09:20:21.010910933Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 485720 compact_task_statistics CompactTaskStatistics { total_file_count: 102, total_key_count: 140817, total_file_size: 12835105, total_uncompressed_file_size: 12819877 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 234704720, enable fast runner: false, Compaction task id: 485720, group-id: 2, type: Dynamic, target level: 0, target sub level: 7091582499749888 watermark: 7097077759868928, target_file_size: 33554432, splits: 1
Input:
Level 0 ["[id: 612466, obj_id: 612466 43KB stale_ratio 0]", "[id: 612456, obj_id: 612456 45KB stale_ratio 0]"]
Level 0 ["[id: 612465, obj_id: 612465 45KB stale_ratio 0]", "[id: 612455, obj_id: 612455 43KB stale_ratio 0]"]
Level 0 ["[id: 612406, obj_id: 612406 1034KB stale_ratio 5]"]
Level 0 ["[id: 612388, obj_id: 612388 2540KB stale_ratio 2]"]
Level 0 ["[id: 612262, obj_id: 612262 54KB stale_ratio 0]", "[id: 612269, obj_id: 612269 58KB stale_ratio 0]"]
Level 0 ["[id: 612259, obj_id: 612259 56KB stale_ratio 0]", "[id: 612270, obj_id: 612270 58KB stale_ratio 0]"]
Level 0 ["[id: 612258, obj_id: 612258 54KB stale_ratio 0]", "[id: 612268, obj_id: 612268 61KB stale_ratio 0]"]
Level 0 ["[id: 612257, obj_id: 612257 54KB stale_ratio 0]", "[id: 612267, obj_id: 612267 55KB stale_ratio 0]"]
Level 0 ["[id: 612256, obj_id: 612256 52KB stale_ratio 0]", "[id: 612246, obj_id: 612246 58KB stale_ratio 0]"]
Level 0 ["[id: 612255, obj_id: 612255 56KB stale_ratio 0]", "[id: 612245, obj_id: 612245 56KB stale_ratio 0]"]
Level 0 ["[id: 612254, obj_id: 612254 56KB stale_ratio 0]", "[id: 612244, obj_id: 612244 56KB stale_ratio 0]"]
Level 0 ["[id: 612253, obj_id: 612253 52KB stale_ratio 0]", "[id: 612243, obj_id: 612243 58KB stale_ratio 0]"]
Level 0 ["[id: 612250, obj_id: 612250 45KB stale_ratio 0]", "[id: 612242, obj_id: 612242 48KB stale_ratio 0]"]
Level 0 ["[id: 612252, obj_id: 612252 56KB stale_ratio 0]", "[id: 612241, obj_id: 612241 55KB stale_ratio 0]"]
Level 0 ["[id: 612249, obj_id: 612249 52KB stale_ratio 0]", "[id: 612240, obj_id: 612240 56KB stale_ratio 0]"]
Level 0 ["[id: 612251, obj_id: 612251 51KB stale_ratio 0]", "[id: 612239, obj_id: 612239 57KB stale_ratio 0]"]
Level 0 ["[id: 612238, obj_id: 612238 57KB stale_ratio 0]", "[id: 612248, obj_id: 612248 54KB stale_ratio 0]"]
Level 0 ["[id: 612237, obj_id: 612237 49KB stale_ratio 0]", "[id: 612247, obj_id: 612247 56KB stale_ratio 0]"]
Level 0 ["[id: 612236, obj_id: 612236 54KB stale_ratio 0]", "[id: 612226, obj_id: 612226 50KB stale_ratio 0]"]
Level 0 ["[id: 612235, obj_id: 612235 53KB stale_ratio 0]", "[id: 612225, obj_id: 612225 52KB stale_ratio 0]"]
Level 0 ["[id: 612234, obj_id: 612234 55KB stale_ratio 0]", "[id: 612224, obj_id: 612224 50KB stale_ratio 0]"]
Level 0 ["[id: 612233, obj_id: 612233 55KB stale_ratio 0]", "[id: 612223, obj_id: 612223 55KB stale_ratio 0]"]
Level 0 ["[id: 612232, obj_id: 612232 56KB stale_ratio 0]", "[id: 612222, obj_id: 612222 52KB stale_ratio 0]"]
Level 0 ["[id: 612231, obj_id: 612231 55KB stale_ratio 0]", "[id: 612221, obj_id: 612221 53KB stale_ratio 0]"]
Level 0 ["[id: 612220, obj_id: 612220 51KB stale_ratio 0]", "[id: 612230, obj_id: 612230 54KB stale_ratio 0]"]
Level 0 ["[id: 612219, obj_id: 612219 52KB stale_ratio 0]", "[id: 612229, obj_id: 612229 53KB stale_ratio 0]"]
Level 0 ["[id: 612218, obj_id: 612218 50KB stale_ratio 0]", "[id: 612228, obj_id: 612228 60KB stale_ratio 0]"]
Level 0 ["[id: 612217, obj_id: 612217 50KB stale_ratio 0]", "[id: 612227, obj_id: 612227 50KB stale_ratio 0]"]
Level 0 ["[id: 612206, obj_id: 612206 49KB stale_ratio 0]", "[id: 612216, obj_id: 612216 51KB stale_ratio 0]"]
Level 0 ["[id: 612205, obj_id: 612205 50KB stale_ratio 0]", "[id: 612215, obj_id: 612215 51KB stale_ratio 0]"]
Level 0 ["[id: 612204, obj_id: 612204 57KB stale_ratio 0]", "[id: 612214, obj_id: 612214 61KB stale_ratio 0]"]
Level 0 ["[id: 612203, obj_id: 612203 37KB stale_ratio 0]", "[id: 612212, obj_id: 612212 47KB stale_ratio 0]"]
Level 0 ["[id: 612202, obj_id: 612202 34KB stale_ratio 0]", "[id: 612211, obj_id: 612211 43KB stale_ratio 0]"]
Level 0 ["[id: 612201, obj_id: 612201 36KB stale_ratio 0]", "[id: 612210, obj_id: 612210 39KB stale_ratio 0]"]
Level 0 ["[id: 612200, obj_id: 612200 51KB stale_ratio 0]", "[id: 612208, obj_id: 612208 48KB stale_ratio 0]"]
Level 0 ["[id: 612199, obj_id: 612199 44KB stale_ratio 0]", "[id: 612213, obj_id: 612213 55KB stale_ratio 0]"]
Level 0 ["[id: 612198, obj_id: 612198 42KB stale_ratio 0]", "[id: 612209, obj_id: 612209 47KB stale_ratio 0]"]
Level 0 ["[id: 612197, obj_id: 612197 38KB stale_ratio 0]", "[id: 612207, obj_id: 612207 37KB stale_ratio 0]"]
Level 0 ["[id: 612186, obj_id: 612186 34KB stale_ratio 0]", "[id: 612196, obj_id: 612196 36KB stale_ratio 0]"]
Level 0 ["[id: 612185, obj_id: 612185 35KB stale_ratio 0]", "[id: 612195, obj_id: 612195 36KB stale_ratio 0]"]
Level 0 ["[id: 612184, obj_id: 612184 37KB stale_ratio 0]", "[id: 612194, obj_id: 612194 36KB stale_ratio 0]"]
Level 0 ["[id: 612183, obj_id: 612183 36KB stale_ratio 0]", "[id: 612193, obj_id: 612193 42KB stale_ratio 0]"]
Level 0 ["[id: 612182, obj_id: 612182 36KB stale_ratio 0]", "[id: 612191, obj_id: 612191 40KB stale_ratio 0]"]
Level 0 ["[id: 612181, obj_id: 612181 38KB stale_ratio 0]", "[id: 612190, obj_id: 612190 38KB stale_ratio 0]"]
Level 0 ["[id: 612178, obj_id: 612178 35KB stale_ratio 0]", "[id: 612188, obj_id: 612188 39KB stale_ratio 0]"]
Level 0 ["[id: 612177, obj_id: 612177 36KB stale_ratio 0]", "[id: 612192, obj_id: 612192 71KB stale_ratio 0]"]
Level 0 ["[id: 612179, obj_id: 612179 754KB stale_ratio 0]", "[id: 612189, obj_id: 612189 1576KB stale_ratio 0]"]
Level 0 ["[id: 612159, obj_id: 612159 32KB stale_ratio 0]", "[id: 612169, obj_id: 612169 34KB stale_ratio 0]"]
Level 0 ["[id: 612168, obj_id: 612168 34KB stale_ratio 0]", "[id: 612157, obj_id: 612157 32KB stale_ratio 0]"]
Level 0 ["[id: 612158, obj_id: 612158 32KB stale_ratio 0]", "[id: 612167, obj_id: 612167 34KB stale_ratio 0]"]
Level 0 ["[id: 612146, obj_id: 612146 85KB stale_ratio 0]", "[id: 612155, obj_id: 612155 160KB stale_ratio 0]"]
Level 0 ["[id: 612144, obj_id: 612144 520KB stale_ratio 0]", "[id: 612154, obj_id: 612154 1307KB stale_ratio 0]"]
Dropped table_ids: {289, 264, 287, 288, 240, 239, 241, 269, 262, 238, 265, 286, 270, 263, 275, 277}

thread 'rw-compaction' panicked at /risingwave/src/storage/hummock_sdk/src/key.rs:1089:21:
key UserKey { 10, TableKey { 000000017065727369737465096e743a2f2f707269096d652d7472616469096e672d71612f707509626c69632d61706109632d636f6d6d616e09642f66786d5f6e7509766f5f726174655f0961737369676e6d65096e742d6a736f6e0007 } } epoch EpochWithGap(7091599306850304) >= prev epoch EpochWithGap(7091599306850304)
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:72:14
   2: risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe_multi_version
             at ./risingwave/src/storage/hummock_sdk/src/key.rs:1089:21
   3: risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe
             at ./risingwave/src/storage/hummock_sdk/src/key.rs:1038:9
   4: risingwave_storage::hummock::compactor::compactor_runner::compact_and_build_sst::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:662:31
   5: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
   6: risingwave_storage::hummock::compactor::Compactor::compact_key_range_impl::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/mod.rs:262:10
   7: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
   8: risingwave_storage::hummock::compactor::Compactor::compact_key_range::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/mod.rs:171:18
   9: risingwave_storage::hummock::compactor::compactor_runner::CompactorRunner::run::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:152:14
  10: risingwave_storage::hummock::compactor::compactor_runner::compact::{{closure}}::{{closure}}
             at ./risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:489:18
  11: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  12: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  13: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  14: await_tree::root::TreeRoot::instrument::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/root.rs:43:34
  15: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
  16: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  22: std::panicking::try::do_call
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  23: std::panicking::try
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  24: std::panic::catch_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  25: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  26: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  27: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  28: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  29: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  30: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.
rs:639:22
  31: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  32: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  33: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  36: tokio::runtime::context::scoped::Scoped<T>::set
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  37: tokio::runtime::context::set_scheduler::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  38: std::thread::local::LocalKey<T>::try_with
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:284:16
  39: std::thread::local::LocalKey<T>::with
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:260:9
  40: tokio::runtime::context::set_scheduler
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17
  41: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  42: tokio::runtime::context::runtime::enter_runtime
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  43: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  44: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  45: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  46: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  47: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  48: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  49: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  50: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  51: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  52: std::panicking::try::do_call
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  53: std::panicking::try
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  54: std::panic::catch_unwind
             at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  55: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  56: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  57: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  58: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  59: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  60: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  61: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  62: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13

*** await tree context of current task ***

Compaction Task 485720 Split 0  [754.074ms]
  compact [754.074ms]
    compact_and_build_sst [754.074ms]  <== current

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
opentelemetry tracing will be exported to `http://host1:5690/` if enabled
launching `compactor`
2024-09-05T09:20:22.183729803Z  INFO risingwave_rt::panic_hook: coredump on panic is likely to be enabled limit=1844674407370955161