search

MaterializeInc / materialize

The Cloud Operational Data Store: use SQL to transform, deliver, and act on fast-changing data.
https://materialize.com
Other
5.71k stars 466 forks source link

0dt upgrade: thread 'tokio:work-11' panicked at src/txn-wal/src/lib.rs:510:53: data should not be closed #28220

Closed def- closed 1 month ago

def- commented 1 month ago

What version of Materialize are you using?

15119a9478 (Pull Request #28191)

What is the issue?

Seen in https://buildkite.com/materialize/nightly/builds/8533#0190ab78-992b-482c-a715-b97127ac88b2:

thread 'tokio:work-11' panicked at /var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/lib.rs:510:53:
data should not be closed
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
   2: core::panicking::panic_display
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:197:5
   3: core::panicking::panic_str
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:172:5
   4: core::option::expect_failed
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/option.rs:1995:5
   5: <core::option::Option<mz_repr::timestamp::Timestamp>>::expect
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/option.rs:896:21
   6: mz_txn_wal::apply_caa::<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>::{closure#0}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/lib.rs:510:31
   7: mz_txn_wal::apply_caa::<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/lib.rs:441:1
   8: <mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}::{closure#0}::{closure#0}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/txns.rs:587:34
   9: <futures_util::stream::futures_unordered::FuturesUnordered<<mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}::{closure#0}::{closure#0}::{closure#0}> as futures_core::stream::Stream>::poll_next
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/futures_unordered/mod.rs:518:17
  10: <futures_util::stream::stream::collect::Collect<futures_util::stream::futures_unordered::FuturesUnordered<<mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}::{closure#0}::{closure#0}::{closure#0}>, alloc::vec::Vec<(mz_persist_client::write::WriteHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>, alloc::vec::Vec<(alloc::vec::Vec<u8>, ([u8; 8], mz_persist_types::ShardId))>)>> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/collect.rs:50:26
  11: <mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}::{closure#0}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/txns.rs:603:63
  12: <mz_txn_wal::metrics::InfallibleOpMetrics>::run::<mz_txn_wal::txns::Tidy, <mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}::{closure#0}::{closure#0}>::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/metrics.rs:151:22
  13: <mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/txns.rs:618:10
  14: <mz_txn_wal::txns::TxnsHandle<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>>::apply_le::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/txns.rs:538:5
  15: <mz_txn_wal::txn_write::TxnApply<mz_repr::timestamp::Timestamp>>::apply::<mz_storage_types::sources::SourceData, (), i64, mz_storage_client::controller::PersistEpoch, mz_storage_types::controller::TxnsCodecRow>::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/txn-wal/src/txn_write.rs:365:42
  16: <mz_storage_controller::persist_handles::TxnsTableWorker<mz_repr::timestamp::Timestamp>>::append::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/storage-controller/src/persist_handles.rs:395:75
  17: <tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::TxnsTableWorker<mz_repr::timestamp::Timestamp>>::append::{closure#0}> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  18: <mz_storage_controller::persist_handles::TxnsTableWorker<mz_repr::timestamp::Timestamp>>::run::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/storage-controller/src/persist_handles.rs:251:26
  19: <mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0906b3a3f1b0600d6-1/materialize/test/src/storage-controller/src/persist_handles.rs:145:28
  20: <tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  21: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
  22: <tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>>>>::with_mut::<core::task::poll::Poll<()>, <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
  23: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:30
  24: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
  25: <core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
  26: std::panicking::try::do_call::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}>, core::task::poll::Poll<()>>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  27: std::panicking::try::<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}>>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  28: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}>, core::task::poll::Poll<()>>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  29: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
  30: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll_inner
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
  31: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
  32: tokio::runtime::task::raw::poll::<tracing::instrument::Instrumented<<mz_storage_controller::persist_handles::PersistTableWriteWorker<mz_repr::timestamp::Timestamp>>::new_txns::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:276:5
  33: <tokio::runtime::task::raw::RawTask>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
  34: <tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:400:9
  35: <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:639:22
  36: tokio::runtime::coop::with_budget::<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core>, ()>, <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:107:5
  37: tokio::runtime::coop::budget::<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core>, ()>, <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:73:5
  38: <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  39: <tokio::runtime::scheduler::multi_thread::worker::Context>::run
  40: tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  41: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}, ()>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/scoped.rs:40:9
  42: tokio::runtime::context::set_scheduler::<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context.rs:176:26
  43: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::try_with::<tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>::{closure#0}, ()>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16
  44: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>::{closure#0}, ()>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9
  45: tokio::runtime::context::set_scheduler::<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context.rs:176:17
  46: tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  47: tokio::runtime::context::runtime::enter_runtime::<tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}, ()>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/runtime.rs:65:16
  48: tokio::runtime::scheduler::multi_thread::worker::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  49: <tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  50: <tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/task.rs:42:21
  51: <tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  52: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
  53: <tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>>>>::with_mut::<core::task::poll::Poll<()>, <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
  54: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:30
  55: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
  56: <core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
  57: std::panicking::try::do_call::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}>, core::task::poll::Poll<()>>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  58: std::panicking::try::<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}>>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  59: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}>, core::task::poll::Poll<()>>
             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  60: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
  61: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll_inner
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
  62: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
  63: <tokio::runtime::task::raw::RawTask>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
  64: <tokio::runtime::task::UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule>>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:437:9
  65: <tokio::runtime::blocking::pool::Task>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:159:9
  66: <tokio::runtime::blocking::pool::Inner>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:513:17
  67: <tokio::runtime::blocking::pool::Spawner>::spawn_thread::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:471:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

ci-regexp: panicked at .* data should not be closed

def- commented 1 month ago

Just noticed, this happened after https://github.com/MaterializeInc/materialize/issues/28216, so maybe that is just the root cause.

benesch commented 1 month ago

This is also possibly fixed by https://github.com/MaterializeInc/materialize/pull/28324.🤞🏽

def- commented 1 month ago

Seems so, haven't seen this fail with #28324 in https://buildkite.com/materialize/nightly/builds/8628

def- commented 1 month ago

My bad, occurred again: https://ci-failures.dev.materialize.com/?key=test-failures&tfFilters=%5B%7B%22id%22%3A%22issue%22%2C%22value%22%3A%22materialize%2F28220%22%7D%5D Buildkite: https://buildkite.com/materialize/nightly/builds/8675 Code state included your PR

aljoscha commented 1 month ago

Forwarding a slack convo:

My current thinking is that this is caused by the non-existence of this line: https://github.com/MaterializeInc/materialize/blame/ac9a5555cefd9c024dc42098e9f164a32cf51b70/src/storage-client/src/storage_collections.rs#L2436.

We only taught StorageCollections to not do shard finalization in v0.108.x (https://github.com/MaterializeInc/materialize/pull/28118), and the test is trying to upgrade from v0.106.x to v0.107.x. I haven’t yet 100% confirmed this, because I didn’t yet figure out how to enable debug logs for a given crate in platform checks.

I have a draft PR for checking this theory: https://github.com/MaterializeInc/materialize/pull/28432

def- commented 1 month ago

You were right, this hasn't occurred again after doing upgrades from X-3 instead of X-4. Since we're a week further, I'll switch us back to X-4 upgrades and close this if it doesn't reoccur.