search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.02k stars 577 forks source link

panic: pk should be in the cache #11170

Closed fuyufjh closed 1 year ago

fuyufjh commented 1 year ago

Describe the bug

Node failed and cannot recover.

image image

Error message/log

thread 'risingwave-streaming-actor' panicked at 'pk [0, 128, 0, 0, 0, 0, 91, 83, 241] should be in the cache', src/stream/src/executor/managed_state/join/join_entry_state.rs:62:13
  19:     0x562b204eca43 - core::panicking::panic_fmt::h1cc8517ae3119d0e
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panicking.rs:67:14
  20:     0x562b260f2fac - risingwave_stream::executor::managed_state::join::join_entry_state::JoinEntryState::remove::h1d4a5e044872e8d6
                               at /risingwave/src/stream/src/executor/managed_state/join/join_entry_state.rs:62:13
  21:     0x562b21d9b395 - risingwave_stream::executor::managed_state::join::JoinHashMap<K,S>::delete_row::h3502b9828aeffb42
                               at /risingwave/src/stream/src/executor/managed_state/join/mod.rs:543:13
  22:     0x562b22939867 - risingwave_stream::executor::hash_join::HashJoinExecutor<K,S,_>::eq_join_oneside::{{closure}}::h5a664e62be7314e9
                               at /risingwave/src/stream/src/executor/hash_join.rs:1265:29
  23:     0x562b2256cc11 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::ha78122221158ad6c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33
  24:     0x562b2256cc11 - risingwave_stream::executor::hash_join::HashJoinExecutor<K,S,_>::into_stream::{{closure}}::h141f3a7cc75e3cb5
                               at /risingwave/src/stream/src/executor/hash_join.rs:706:5
full stack backtrace: ``` 0: 0x562b264223ec - std::backtrace_rs::backtrace::libunwind::trace::hc317256c0daecb22 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5 1: 0x562b264223ec - std::backtrace_rs::backtrace::trace_unsynchronized::h85008105ccd95ba6 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x562b264223ec - std::sys_common::backtrace::_print_fmt::hb6e7cbe31fecc91c at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:65:5 3: 0x562b264223ec - ::fmt::h53097d3751045778 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:44:22 4: 0x562b204efe1f - core::fmt::rt::Argument::fmt::hdbe0b5dfb2c09e86 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/fmt/rt.rs:138:9 5: 0x562b204efe1f - core::fmt::write::h4ae17c68778671a0 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/fmt/mod.rs:1094:21 6: 0x562b2641d2ef - std::io::Write::write_fmt::hfd7c55e775c6dc60 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/io/mod.rs:1713:15 7: 0x562b264221f5 - std::sys_common::backtrace::_print::h81eabb0036b692a0 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:47:5 8: 0x562b264221f5 - std::sys_common::backtrace::print::h1b1f4d1a23fac3fd at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:34:9 9: 0x562b26423767 - std::panicking::default_hook::{{closure}}::hc17d2bd8f3da442c 10: 0x562b26423574 - std::panicking::default_hook::h3a32a9336113ba96 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:288:9 11: 0x562b25d00d9b - as core::ops::function::Fn>::call::h992128371750157e at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1999:9 12: 0x562b25d00d9b - risingwave_rt::set_panic_hook::{{closure}}::hb250041294570678 at /risingwave/src/utils/runtime/src/lib.rs:126:9 13: 0x562b25d00d9b - std::panicking::update_hook::{{closure}}::heec6c83e7aaac63d at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:234:47 14: 0x562b26423d82 - as core::ops::function::Fn>::call::h4413820818220cb9 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1999:9 15: 0x562b26423d82 - std::panicking::rust_panic_with_hook::h90e4d15277396259 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:709:13 16: 0x562b26423b24 - std::panicking::begin_panic_handler::{{closure}}::h2b743ea215b7c408 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:597:13 17: 0x562b264227c6 - std::sys_common::backtrace::__rust_end_short_backtrace::h895d390908a90650 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:151:18 18: 0x562b264238b2 - rust_begin_unwind at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:593:5 19: 0x562b204eca43 - core::panicking::panic_fmt::h1cc8517ae3119d0e at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panicking.rs:67:14 20: 0x562b260f2fac - risingwave_stream::executor::managed_state::join::join_entry_state::JoinEntryState::remove::h1d4a5e044872e8d6 at /risingwave/src/stream/src/executor/managed_state/join/join_entry_state.rs:62:13 21: 0x562b21d9b395 - risingwave_stream::executor::managed_state::join::JoinHashMap::delete_row::h3502b9828aeffb42 at /risingwave/src/stream/src/executor/managed_state/join/mod.rs:543:13 22: 0x562b22939867 - risingwave_stream::executor::hash_join::HashJoinExecutor::eq_join_oneside::{{closure}}::h5a664e62be7314e9 at /risingwave/src/stream/src/executor/hash_join.rs:1265:29 23: 0x562b2256cc11 - as futures_core::stream::Stream>::poll_next::ha78122221158ad6c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 24: 0x562b2256cc11 - risingwave_stream::executor::hash_join::HashJoinExecutor::into_stream::{{closure}}::h141f3a7cc75e3cb5 at /risingwave/src/stream/src/executor/hash_join.rs:706:5 25: 0x562b2256cc11 - as futures_core::stream::Stream>::poll_next::h2da82f4731396d11 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 26: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 27: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hbd34594d356fceb0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 28: 0x562b261b7835 - futures_util::stream::stream::StreamExt::poll_next_unpin::h25c3d5782f8e67ab at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 29: 0x562b261b7835 - as core::future::future::Future>::poll::h842f8de74c26be06 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 30: 0x562b261b7835 - as core::future::future::Future>::poll::h186de3f02b0c4bce at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23 31: 0x562b261de04b - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h9b17c8e9a602cac0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:100:10 32: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hbd070814821a9b0d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 33: 0x562b261de04b - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h159097dd9c049e0b at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1 34: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hf39fbd43b290b768 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 35: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::hf9de9e711371e55c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 36: 0x562b261c5021 - futures_util::stream::stream::StreamExt::poll_next_unpin::h31fa6da45b01cfa1 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 37: 0x562b261c5021 - as core::future::future::Future>::poll::hdb3e856a0137c2ae at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 38: 0x562b261c5021 - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::ha145565329ad0b7d at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44 39: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::h2bc94e71da865279 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 40: 0x562b26114f57 - as futures_core::stream::Stream>::poll_next::h171d35002a65a43a at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 41: 0x562b26114f57 - ::try_poll_next::h9ca00db64ddc9014 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 42: 0x562b26114f57 - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::hc472de35d0dfc0f0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9 43: 0x562b26114f57 - as core::future::future::Future>::poll::h84d32bf406f89133 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:21 44: 0x562b26114f57 - as core::future::future::Future>::poll::{{closure}}::h9ab164452a97306f at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:348:35 45: 0x562b26114f57 - tokio::task::task_local::LocalKey::scope_inner::h8368d7ee110d04b2 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:233:19 46: 0x562b26114f57 - as core::future::future::Future>::poll::he9c9124632006336 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:345:13 47: 0x562b261cb078 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h43cf4387c842182c at /risingwave/src/common/src/util/epoch.rs:181:47 48: 0x562b261cb078 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h9752788be45d61f1 at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59 49: 0x562b261cb078 - as futures_core::stream::Stream>::poll_next::h70041975314de6a8 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 50: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::haa0d269156ec3a7d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 51: 0x562b261d3041 - futures_util::stream::stream::StreamExt::poll_next_unpin::h01baebef87952c4b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 52: 0x562b261d3041 - as core::future::future::Future>::poll::h7964445690b6a715 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 53: 0x562b261d3041 - as core::future::future::Future>::poll::h11f18a9dca2b7a60 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 54: 0x562b261d3041 - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::h4342bdb75bcc08f0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:46:69 55: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::ha3033574b0287850 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 56: 0x562b260caf11 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 57: 0x562b260caf11 - ::try_poll_next::he1e3656e309c0774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 58: 0x562b261d7bb6 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 59: 0x562b261d7bb6 - risingwave_stream::executor::filter::FilterExecutor::execute_inner::{{closure}}::h293417c2d38b08ea at /risingwave/src/stream/src/executor/filter.rs:166:5 60: 0x562b261d7bb6 - as futures_core::stream::Stream>::poll_next::hcd09b443e48fa615 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 61: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 62: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hbd34594d356fceb0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 63: 0x562b261b7835 - futures_util::stream::stream::StreamExt::poll_next_unpin::h25c3d5782f8e67ab at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 64: 0x562b261b7835 - as core::future::future::Future>::poll::h842f8de74c26be06 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 65: 0x562b261b7835 - as core::future::future::Future>::poll::h186de3f02b0c4bce at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23 66: 0x562b261de04b - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h9b17c8e9a602cac0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:100:10 67: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hbd070814821a9b0d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 68: 0x562b261de04b - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h159097dd9c049e0b at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1 69: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hf39fbd43b290b768 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 70: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::hf9de9e711371e55c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 71: 0x562b261c5021 - futures_util::stream::stream::StreamExt::poll_next_unpin::h31fa6da45b01cfa1 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 72: 0x562b261c5021 - as core::future::future::Future>::poll::hdb3e856a0137c2ae at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 73: 0x562b261c5021 - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::ha145565329ad0b7d at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44 74: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::h2bc94e71da865279 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 75: 0x562b26114f57 - as futures_core::stream::Stream>::poll_next::h171d35002a65a43a at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 76: 0x562b26114f57 - ::try_poll_next::h9ca00db64ddc9014 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 77: 0x562b26114f57 - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::hc472de35d0dfc0f0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9 78: 0x562b26114f57 - as core::future::future::Future>::poll::h84d32bf406f89133 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:21 79: 0x562b26114f57 - as core::future::future::Future>::poll::{{closure}}::h9ab164452a97306f at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:348:35 80: 0x562b26114f57 - tokio::task::task_local::LocalKey::scope_inner::h8368d7ee110d04b2 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:233:19 81: 0x562b26114f57 - as core::future::future::Future>::poll::he9c9124632006336 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:345:13 82: 0x562b261cb078 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h43cf4387c842182c at /risingwave/src/common/src/util/epoch.rs:181:47 83: 0x562b261cb078 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h9752788be45d61f1 at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59 84: 0x562b261cb078 - as futures_core::stream::Stream>::poll_next::h70041975314de6a8 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 85: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::haa0d269156ec3a7d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 86: 0x562b261d3041 - futures_util::stream::stream::StreamExt::poll_next_unpin::h01baebef87952c4b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 87: 0x562b261d3041 - as core::future::future::Future>::poll::h7964445690b6a715 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 88: 0x562b261d3041 - as core::future::future::Future>::poll::h11f18a9dca2b7a60 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 89: 0x562b261d3041 - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::h4342bdb75bcc08f0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:46:69 90: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::ha3033574b0287850 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 91: 0x562b260caf11 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 92: 0x562b260caf11 - ::try_poll_next::he1e3656e309c0774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 93: 0x562b261d43e5 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 94: 0x562b261d43e5 - risingwave_stream::executor::project::Inner::execute::{{closure}}::h9f8d39a78f1bfc1c at /risingwave/src/stream/src/executor/project.rs:169:5 95: 0x562b261d43e5 - as futures_core::stream::Stream>::poll_next::haf1d369debfa0d41 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 96: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 97: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hbd34594d356fceb0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 98: 0x562b261b7835 - futures_util::stream::stream::StreamExt::poll_next_unpin::h25c3d5782f8e67ab at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 99: 0x562b261b7835 - as core::future::future::Future>::poll::h842f8de74c26be06 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 100: 0x562b261b7835 - as core::future::future::Future>::poll::h186de3f02b0c4bce at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23 101: 0x562b261de04b - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h9b17c8e9a602cac0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:100:10 102: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hbd070814821a9b0d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 103: 0x562b261de04b - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h159097dd9c049e0b at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1 104: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hf39fbd43b290b768 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 105: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::hf9de9e711371e55c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 106: 0x562b261c5021 - futures_util::stream::stream::StreamExt::poll_next_unpin::h31fa6da45b01cfa1 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 107: 0x562b261c5021 - as core::future::future::Future>::poll::hdb3e856a0137c2ae at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 108: 0x562b261c5021 - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::ha145565329ad0b7d at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44 109: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::h2bc94e71da865279 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 110: 0x562b26114f57 - as futures_core::stream::Stream>::poll_next::h171d35002a65a43a at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 111: 0x562b26114f57 - ::try_poll_next::h9ca00db64ddc9014 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 112: 0x562b26114f57 - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::hc472de35d0dfc0f0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9 113: 0x562b26114f57 - as core::future::future::Future>::poll::h84d32bf406f89133 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:21 114: 0x562b26114f57 - as core::future::future::Future>::poll::{{closure}}::h9ab164452a97306f at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:348:35 115: 0x562b26114f57 - tokio::task::task_local::LocalKey::scope_inner::h8368d7ee110d04b2 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:233:19 116: 0x562b26114f57 - as core::future::future::Future>::poll::he9c9124632006336 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:345:13 117: 0x562b261cb078 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h43cf4387c842182c at /risingwave/src/common/src/util/epoch.rs:181:47 118: 0x562b261cb078 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h9752788be45d61f1 at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59 119: 0x562b261cb078 - as futures_core::stream::Stream>::poll_next::h70041975314de6a8 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 120: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::haa0d269156ec3a7d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 121: 0x562b261d3041 - futures_util::stream::stream::StreamExt::poll_next_unpin::h01baebef87952c4b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 122: 0x562b261d3041 - as core::future::future::Future>::poll::h7964445690b6a715 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 123: 0x562b261d3041 - as core::future::future::Future>::poll::h11f18a9dca2b7a60 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 124: 0x562b261d3041 - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::h4342bdb75bcc08f0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:46:69 125: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::ha3033574b0287850 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 126: 0x562b260caf11 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 127: 0x562b260caf11 - ::try_poll_next::he1e3656e309c0774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 128: 0x562b22634719 - as futures_core::stream::Stream>::poll_next::he85fab4172d6f951 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 129: 0x562b22634719 - risingwave_stream::executor::mview::materialize::MaterializeExecutor::execute_inner::{{closure}}::h97bbf709b435dfd6 at /risingwave/src/stream/src/executor/mview/materialize.rs:120:5 130: 0x562b22634719 - as futures_core::stream::Stream>::poll_next::h8f2419ff0cd327e8 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 131: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 132: 0x562b261b7835 - as futures_core::stream::Stream>::poll_next::hbd34594d356fceb0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 133: 0x562b261b7835 - futures_util::stream::stream::StreamExt::poll_next_unpin::h25c3d5782f8e67ab at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 134: 0x562b261b7835 - as core::future::future::Future>::poll::h842f8de74c26be06 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 135: 0x562b261b7835 - as core::future::future::Future>::poll::h186de3f02b0c4bce at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23 136: 0x562b261de04b - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h9b17c8e9a602cac0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:100:10 137: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hbd070814821a9b0d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 138: 0x562b261de04b - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h159097dd9c049e0b at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1 139: 0x562b261de04b - as futures_core::stream::Stream>::poll_next::hf39fbd43b290b768 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 140: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::hf9de9e711371e55c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 141: 0x562b261c5021 - futures_util::stream::stream::StreamExt::poll_next_unpin::h31fa6da45b01cfa1 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 142: 0x562b261c5021 - as core::future::future::Future>::poll::hdb3e856a0137c2ae at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 143: 0x562b261c5021 - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::ha145565329ad0b7d at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44 144: 0x562b261c5021 - as futures_core::stream::Stream>::poll_next::h2bc94e71da865279 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 145: 0x562b26114f57 - as futures_core::stream::Stream>::poll_next::h171d35002a65a43a at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 146: 0x562b26114f57 - ::try_poll_next::h9ca00db64ddc9014 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 147: 0x562b26114f57 - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::hc472de35d0dfc0f0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9 148: 0x562b26114f57 - as core::future::future::Future>::poll::h84d32bf406f89133 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:21 149: 0x562b26114f57 - as core::future::future::Future>::poll::{{closure}}::h9ab164452a97306f at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:348:35 150: 0x562b26114f57 - tokio::task::task_local::LocalKey::scope_inner::h8368d7ee110d04b2 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:233:19 151: 0x562b26114f57 - as core::future::future::Future>::poll::he9c9124632006336 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:345:13 152: 0x562b261cb078 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h43cf4387c842182c at /risingwave/src/common/src/util/epoch.rs:181:47 153: 0x562b261cb078 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h9752788be45d61f1 at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59 154: 0x562b261cb078 - as futures_core::stream::Stream>::poll_next::h70041975314de6a8 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 155: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::haa0d269156ec3a7d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 156: 0x562b261d3041 - futures_util::stream::stream::StreamExt::poll_next_unpin::h01baebef87952c4b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 157: 0x562b261d3041 - as core::future::future::Future>::poll::h7964445690b6a715 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21 158: 0x562b261d3041 - as core::future::future::Future>::poll::h11f18a9dca2b7a60 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 159: 0x562b261d3041 - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::h4342bdb75bcc08f0 at /risingwave/src/stream/src/executor/wrapper/trace.rs:46:69 160: 0x562b261d3041 - as futures_core::stream::Stream>::poll_next::ha3033574b0287850 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 161: 0x562b260caf11 - as futures_core::stream::Stream>::poll_next::hcfc138febea10774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 162: 0x562b260caf11 - ::try_poll_next::he1e3656e309c0774 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 163: 0x562b226aa81b - as futures_core::stream::Stream>::poll_next::he85fab4172d6f951 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 164: 0x562b226aa81b - ::execute::{{closure}}::h3f1504f37b124f2e at /risingwave/src/stream/src/executor/dispatch.rs:258:9 165: 0x562b226aa81b - as futures_core::stream::Stream>::poll_next::hd1d51dc439539bd5 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.6/src/lib.rs:463:33 166: 0x562b221baaf9 - as futures_core::stream::Stream>::poll_next::h6b6002509bf1d008 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 167: 0x562b221baaf9 - <&mut S as futures_core::stream::Stream>::poll_next::h0f8d373e6e585651 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:104:9 168: 0x562b221baaf9 - as core::future::future::Future>::poll::h53d9f97adaa5c9d0 at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/ab251ad/tokio-stream/src/stream_ext/next.rs:42:9 169: 0x562b221baaf9 - as core::future::future::Future>::poll::haf5aa53b6d29df2c at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/ab251ad/tokio-stream/src/stream_ext/try_next.rs:43:9 170: 0x562b221baaf9 - as core::future::future::Future>::poll::hc4bdecf344b7d021 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 171: 0x562b2225a29a - as core::future::future::Future>::poll::h7e9a54f65156184e at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23 172: 0x562b22c511c4 - risingwave_stream::executor::actor::Actor::run_consumer::{{closure}}::hd16fd93cedf7b013 at /risingwave/src/stream/src/executor/actor.rs:190:18 173: 0x562b22c511c4 - as core::future::future::Future>::poll::h1395f8c915a2313d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/future/maybe_done.rs:68:48 174: 0x562b221b1aeb - risingwave_stream::executor::actor::Actor::run::{{closure}}::{{closure}}::h2a26b04ff5e4f7db at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/macros/join.rs:126:24 175: 0x562b221b1aeb - as core::future::future::Future>::poll::h1ae6aebf69370104 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/future/poll_fn.rs:58:9 176: 0x562b21e6b44b - risingwave_stream::executor::actor::Actor::run::{{closure}}::h4975df32de79ba60 at /risingwave/src/stream/src/executor/actor.rs:154:9 177: 0x562b21e6b44b - risingwave_stream::task::stream_manager::LocalStreamManagerCore::build_actors::{{closure}}::{{closure}}::hc592fa88c0ad8e8c at /risingwave/src/stream/src/task/stream_manager.rs:652:51 178: 0x562b21e6b44b - as core::future::future::Future>::poll::{{closure}}::hd47096de2757d7b4 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:348:35 179: 0x562b21e6b44b - tokio::task::task_local::LocalKey::scope_inner::hd76df2f02694457a at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:233:19 180: 0x562b21e6b44b - as core::future::future::Future>::poll::h1ec6cb29424e71b9 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/task/task_local.rs:345:13 181: 0x562b224661e1 - await_tree::registry::TreeRoot::instrument::{{closure}}::h5ffc51197c216eab at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/registry.rs:51:45 182: 0x562b224661e1 - as core::future::future::Future>::poll::h50828b1288fc70e4 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/either.rs:109:32 183: 0x562b21fe30eb - as core::future::future::Future>::poll::h990545a805e4dbf3 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.1.0/src/task.rs:2365:19 184: 0x562b2216922a - as core::future::future::Future>::poll::hcd69fe441d54a62b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 185: 0x562b2216922a - tokio::runtime::task::core::Core::poll::{{closure}}::h08cc6fb37a925f47 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17 186: 0x562b2216922a - tokio::loom::std::unsafe_cell::UnsafeCell::with_mut::h7f1bb6f2e185c931 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9 187: 0x562b21eb3749 - tokio::runtime::task::core::Core::poll::h063ce6d8fa43705f at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13 188: 0x562b2217732b - tokio::runtime::task::harness::poll_future::{{closure}}::hb21d9bbbb25be88b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19 189: 0x562b2217732b - as core::ops::function::FnOnce<()>>::call_once::hf7c9d7d8ae5c811f at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9 190: 0x562b2217732b - std::panicking::try::do_call::hb0b806c0cff87777 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40 191: 0x562b2217732b - std::panicking::try::h79f12ad5de7c96f8 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19 192: 0x562b2217732b - std::panic::catch_unwind::h79550cbdc74d49b5 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14 193: 0x562b2217732b - tokio::runtime::task::harness::poll_future::h7ebb8c335c6bea90 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18 194: 0x562b2217732b - tokio::runtime::task::harness::Harness::poll_inner::h180944004fd1d764 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27 195: 0x562b2217732b - tokio::runtime::task::harness::Harness::poll::h1b075f5b81967444 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15 196: 0x562b264b7f5f - tokio::runtime::task::raw::RawTask::poll::he875d76631553c7d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18 197: 0x562b264b7f5f - tokio::runtime::task::LocalNotified::run::h1986ccfed02f1633 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:394:9 198: 0x562b264b7f5f - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h9695960ae749e0c0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:487:26 199: 0x562b264b7f5f - tokio::runtime::coop::with_budget::h6c6833cd6e3d9f17 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:107:5 200: 0x562b264b7f5f - tokio::runtime::coop::budget::h3867becfa41cc4ad at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:73:5 201: 0x562b264b7f5f - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hd99cb567e7166fa1 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:463:9 202: 0x562b264b7493 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h2bde6096a99f054c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:426:24 203: 0x562b264b10dd - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::hb1a44c99c5471db8 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:406:17 204: 0x562b264b10dd - tokio::macros::scoped_tls::ScopedKey::set::h94ba58eeb5a9defa at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/macros/scoped_tls.rs:61:9 205: 0x562b264b6f08 - tokio::runtime::scheduler::multi_thread::worker::run::hd95fdb34818b1f22 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:403:5 206: 0x562b2649f37c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h537ed4cb460a0cc1 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:365:45 207: 0x562b2649f37c - as core::future::future::Future>::poll::h52e926ff39b578fb at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/task.rs:42:21 208: 0x562b2649f37c - as core::future::future::Future>::poll::h627de37005ea65d4 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9 209: 0x562b264a5f6d - tokio::runtime::task::core::Core::poll::{{closure}}::h04f34061caea69da at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17 210: 0x562b264a5f6d - tokio::loom::std::unsafe_cell::UnsafeCell::with_mut::hcd88b3b966ce76cb at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9 211: 0x562b264a5f6d - tokio::runtime::task::core::Core::poll::hd335a52ff4bbd6a0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13 212: 0x562b26487d1f - tokio::runtime::task::harness::poll_future::{{closure}}::h960a64aced35d30c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19 213: 0x562b26487d1f - as core::ops::function::FnOnce<()>>::call_once::ha469073e398f345e at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9 214: 0x562b26487d1f - std::panicking::try::do_call::hf2d2b0597bf0e591 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40 215: 0x562b26487d1f - std::panicking::try::h5f5f4e65fdd22f3e at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19 216: 0x562b26487d1f - std::panic::catch_unwind::h3938c820f22bef32 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14 217: 0x562b26487d1f - tokio::runtime::task::harness::poll_future::hc4dae0f9c169ac3a at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18 218: 0x562b26487d1f - tokio::runtime::task::harness::Harness::poll_inner::ha418d67b1cfe8e49 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27 219: 0x562b26487d1f - tokio::runtime::task::harness::Harness::poll::h0992acae90254291 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15 220: 0x562b26486c5f - tokio::runtime::task::raw::RawTask::poll::he875d76631553c7d at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18 221: 0x562b26486c5f - tokio::runtime::task::UnownedTask::run::h9157f1babe3521d4 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:431:9 222: 0x562b26486c5f - tokio::runtime::blocking::pool::Task::run::h19e9a047ecbf685c at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:159:9 223: 0x562b26486c5f - tokio::runtime::blocking::pool::Inner::run::h83f8f78324d4a61b at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:513:17 224: 0x562b264a6e56 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hba3a8f223bfea8e0 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:471:13 225: 0x562b264a6e56 - std::sys_common::backtrace::__rust_begin_short_backtrace::hea836d8e859467c0 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:135:18 226: 0x562b26493092 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h822b7dc4ea005a3a at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/thread/mod.rs:529:17 227: 0x562b26493092 - as core::ops::function::FnOnce<()>>::call_once::hb96058b256b1901c at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9 228: 0x562b26493092 - std::panicking::try::do_call::h6035b7fd53841e53 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40 229: 0x562b26493092 - std::panicking::try::h64b1422bd2c576e6 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19 230: 0x562b26493092 - std::panic::catch_unwind::h7dedd133532675e2 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14 231: 0x562b26493092 - std::thread::Builder::spawn_unchecked_::{{closure}}::h67b8dabaf583a6bf at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/thread/mod.rs:528:30 232: 0x562b26493092 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h103025125bcaee67 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/ops/function.rs:250:5 233: 0x562b26429af5 - as core::ops::function::FnOnce>::call_once::h0716d70b901d1398 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1985:9 234: 0x562b26429af5 - as core::ops::function::FnOnce>::call_once::h7c11e50091aab2e1 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1985:9 235: 0x562b26429af5 - std::sys::unix::thread::Thread::new::thread_start::h3ae99d1c2785d9c6 at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys/unix/thread.rs:108:17 236: 0x7fc963cabb43 - start_thread at ./nptl/pthread_create.c:442:8 237: 0x7fc963d3da00 - clone3 at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 238: 0x0 - *** await tree context of current task *** Actor 35: `nexmark_q5` [4770.281s] Epoch 4781728322355200 [1.590s] MaterializeExecutor 2300000010 (actor 35, operator 16) [1.590s] ProjectExecutor 230000000F (actor 35, operator 15) [1.590s] FilterExecutor 230000000E (actor 35, operator 14) [1.590s] HashJoinExecutor 230000000D (actor 35, operator 13) [1.590s] <== current [Detached 5] ProjectExecutor 230000000C (actor 35, operator 12) [1.590s] HashAggExecutor 230000000B (actor 35, operator 11) [1.590s] MergeExecutor 230000000A (actor 35, operator 10) [1.590s] RemoteInput (actor 37) [1.590s] RemoteInput (actor 39) [1.590s] LocalInput (actor 38) [1.590s] [Detached 10] LocalInput (actor 41) [1.590s] [Detached 18] RemoteInput (actor 42) [1.590s] ```

To Reproduce

Run longevity-test with

BENCH_SKU="medium-3cn"
BUILDKITE="true"
CI="true"
DURATION="10h"
INTERVAL="30m"
NEXMARK_QUERIES="q1,q2,q3,q4,q5,q7,q8,q9,q10,q14,q15,q16,q17,q18,q20"
RW_VERSION="fa08bec9f3dfea87640ea1d4e88a9860f2bec412-x86_64"
SLACK_CHANNEL="test_notification"
STREAMING_PARALLELISM="3"
max_clusters_allowed="2"
skip_cleanup_on_failure="0"
skip_mvs_from_mvs="1"
skip_view_queries_execution="0"
skipcountmvfromsource="1"
user="eric"

Expected behavior

No response

How did you deploy RisingWave?

K8s (longevity-test pipeline)

The version of RisingWave

main branch

Additional context

BuildKite Job

Logs

Grafana

st1page commented 1 year ago

https://github.com/risingwavelabs/risingwave/issues/11216

st1page commented 1 year ago 
pk [0, 128, 0, 0, 0, 0, 91, 83, 241]
auction id = 91*256*256 + 83*256 +241 = 5985265

I can not imagine how the append-only Agg can give a wrong delete event :hot_face: And the Agg's result table's sanity check does not panic. because had not been committed and the sanity check is in the stateStore::flush.

        └─StreamHashJoin { type: Inner, predicate: window_start = window_start, output: all }
          ├─StreamExchange { dist: HashShard(window_start) }
          │ └─StreamShare { id: 7 }
          │   └─StreamHashAgg [append_only] { group_key: [auction, window_start], aggs: [count] }
stdrc commented 1 year ago

Hard to find the reason due to too few information we can get, I just added some log in #11266, hopefully we may get more info next time🥵

fuyufjh commented 1 year ago

Hard to find the reason due to too few information we can get, I just added some log in #11266, hopefully we may get more info next time🥵

You may try to reproduce by running longevity-test with the parameters in PR description.

Btw, Can we know which query is it?

stdrc commented 1 year ago

Btw, Can we know which query is it?

It's nexmark q5

stdrc commented 1 year ago

You may try to reproduce by running longevity-test with the parameters in PR description.

How much time will it take to reproduce? Or is it just randomly happens?

fuyufjh commented 1 year ago

You may try to reproduce by running longevity-test with the parameters in PR description.

How much time will it take to reproduce? Or is it just randomly happens?

In my test, it took 1h20min to occur, as shown in the metrics graph in PR description.

I don't know whether it's random.

stdrc commented 1 year ago

Ran a longevity test with the same environments for 5 hours, hasn't failed🤣. Seems to be a random bug.

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.