asan: thread 'persist:0005' panicked at src/persist-client/src/internal/state_versions.rs:1072:40: assertion failed: self.inc_batches.insert(x.clone())

[def-]

What version of Materialize are you using?

a9ec35a3ef3a (Pull Request #24888)

What is the issue?

Seen in https://buildkite.com/materialize/tests/builds/74892 with ASan enabled, so different timing (https://github.com/MaterializeInc/materialize/pull/24888):

thread 'persist:0005' panicked at /var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/state_versions.rs:1072:40:
assertion failed: self.inc_batches.insert(x.clone())
   0: rust_begin_unwind
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panicking.rs:127:5
   3: <mz_persist_client::internal::state_versions::ReferencedBlobValidator<mz_repr::timestamp::Timestamp>>::add_inc_blob
   4: <mz_persist_client::internal::state_versions::StateVersionsIter<mz_repr::timestamp::Timestamp>>::next::<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::find_removable_blobs<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}::{closure#0}>::{closure#0}>::{closure#2}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/state_versions.rs:983:48
   5: <mz_persist_client::internal::state_diff::StateDiff<mz_repr::timestamp::Timestamp>>::map_blob_inserts::<<mz_persist_client::internal::state_versions::StateVersionsIter<mz_repr::timestamp::Timestamp>>::next<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::find_removable_blobs<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}::{closure#0}>::{closure#0}>::{closure#2}>
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/state_diff.rs:178:21
   6: <mz_persist_client::internal::state_versions::InspectDiff<mz_repr::timestamp::Timestamp>>::referenced_blob_fn::<<mz_persist_client::internal::state_versions::StateVersionsIter<mz_repr::timestamp::Timestamp>>::next<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::find_removable_blobs<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}::{closure#0}>::{closure#0}>::{closure#2}>
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/state_versions.rs:1039:37
   7: <mz_persist_client::internal::state_versions::StateVersionsIter<mz_repr::timestamp::Timestamp>>::next::<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::find_removable_blobs<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}::{closure#0}>::{closure#0}>
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/state_versions.rs:983:17
   8: <mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::find_removable_blobs::<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}::{closure#0}>
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/gc.rs:518:33
   9: <mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::incrementally_delete_and_truncate::<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}::{closure#0}>::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/gc.rs:431:13
  10: <mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/gc.rs:301:10
  11: <tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::gc_and_truncate::{closure#0}> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  12: <mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0cfc073d36a27689b-1/materialize/tests/src/persist-client/src/internal/gc.rs:164:34
  13: <tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  14: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
  15: <tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>>>>::with_mut::<core::task::poll::Poll<(mz_persist_client::internal::maintenance::RoutineMaintenance, mz_persist_client::internal::gc::GcResults)>, <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
  16: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:13
  17: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
  18: <core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panic/unwind_safe.rs:272:9
  19: std::panicking::try::do_call::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}>, core::task::poll::Poll<(mz_persist_client::internal::maintenance::RoutineMaintenance, mz_persist_client::internal::gc::GcResults)>>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:552:40
  20: __rust_try.llvm.2747044270895621303
  21: std::panicking::try::<core::task::poll::Poll<(mz_persist_client::internal::maintenance::RoutineMaintenance, mz_persist_client::internal::gc::GcResults)>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}>>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:516:19
  22: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#0}>, core::task::poll::Poll<(mz_persist_client::internal::maintenance::RoutineMaintenance, mz_persist_client::internal::gc::GcResults)>>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panic.rs:142:14
  23: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
  24: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll_inner
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
  25: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<<mz_persist_client::internal::gc::GarbageCollector<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>>::new::{closure#1}::{closure#1}>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
  26: <tokio::runtime::task::raw::RawTask>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
  27: <tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:400:9
  28: <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:639:22
  29: tokio::runtime::coop::with_budget::<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core>, ()>, <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:107:5
  30: tokio::runtime::coop::budget::<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core>, ()>, <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:73:5
  31: <tokio::runtime::scheduler::multi_thread::worker::Context>::run_task
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  32: <tokio::runtime::scheduler::multi_thread::worker::Context>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  33: tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  34: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}, ()>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/scoped.rs:40:9
  35: tokio::runtime::context::set_scheduler::<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context.rs:176:26
  36: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::try_with::<tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>::{closure#0}, ()>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/thread/local.rs:270:16
  37: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>::{closure#0}, ()>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/thread/local.rs:246:9
  38: tokio::runtime::context::set_scheduler::<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context.rs:176:17
  39: tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  40: tokio::runtime::context::runtime::enter_runtime::<tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}, ()>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/runtime.rs:65:16
  41: tokio::runtime::scheduler::multi_thread::worker::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  42: <tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  43: <tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/task.rs:42:21
  44: <tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>> as core::future::future::Future>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  45: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
  46: <tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>>>>::with_mut::<core::task::poll::Poll<()>, <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll::{closure#0}>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
  47: <tokio::runtime::task::core::Core<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:13
  48: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
  49: <core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panic/unwind_safe.rs:272:9
  50: std::panicking::try::do_call::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}>, core::task::poll::Poll<()>>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:552:40
  51: __rust_try.llvm.15611257329881442963
  52: std::panicking::try::<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}>>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:516:19
  53: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>::{closure#0}>, core::task::poll::Poll<()>>
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panic.rs:142:14
  54: tokio::runtime::task::harness::poll_future::<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
  55: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll_inner
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
  56: <tokio::runtime::task::harness::Harness<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<<tokio::runtime::scheduler::multi_thread::worker::Launch>::launch::{closure#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
  57: <tokio::runtime::task::raw::RawTask>::poll
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
  58: <tokio::runtime::task::UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule>>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:437:9
  59: <tokio::runtime::blocking::pool::Task>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:159:9
  60: <tokio::runtime::blocking::pool::Inner>::run
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:513:17
  61: <tokio::runtime::blocking::pool::Spawner>::spawn_thread::{closure#0}
             at ./cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:471:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

ci-regexp: assertion failed: self.inc_batches.insert

[benesch]

@danhhz can you help me understand how concerning this assertion failure is?

[danhhz]

Looks like the panic is coming out of some code that is only run under debug_assertions. It's one of those "do a thing the tricky, fast way and the obvious, slow way and make sure they give the same answer" sort of things. If they were giving different answers, that would be quite concerning, but it looks like the failure is actually that an assumption of the obvious, slow way is being violated.

Overall, definitely something for us to look into, but doesn't seem especially concerning yet that this would indicate a bug in prod.