shardtree panic

[AArnott]

I'm getting panics when syncing after merging the latest from main into my app. I've narrowed the problem down to this segment of commits:

03fc64cf Merge pull request #1396 from nuttycom/test/block_spanning_shards 3e090dca Apply suggestions from code review 3bc74b25 zcash_client_sqlite: Improve documentation for test setup functions. e3956c32 zcash_client_sqlite: Fix clippy complaint. ecea9ca1 zcash_client_sqlite: Fix shardtree error caused by pruning after frontier insertion. de66c5b1 zcash_client_sqlite: Add test to reproduce incomplete penultimate subtree error. 3db515e3 zcash_client_sqlite: Allow multi-output fake blocks to send to multiple keys. c88cd179 zcash_client_sqlite: Add multiple output capability to fake compact block construction.

The panic is in the shardtree processing, so it's reinforcing to see that shardtree appears in the list of changes.

Exception thrown at 0x00007FFF7A82F39C in Nerdbank.Zcash.App.Desktop.exe: Microsoft C++ exception: ?? ::st_panic at memory location 0x000000B03F132E28.

Code at failure (incrementalmerkletree-963d685c35fcaf01\8b4b131\shardtree\src\prunable.rs:751):

assert!(new_tree.max_position() >= max_position);

Callstack

    KernelBase.dll!00007fff7a82f39c()   Unknown
    vcruntime140.dll!00007fff6eae5267() Unknown
    [Inline Frame] nerdbank_zcash_rust.dll!panic_unwind::real_imp::panic() Line 353 Rust
    nerdbank_zcash_rust.dll!panic_unwind::__rust_start_panic() Line 108 Rust
    nerdbank_zcash_rust.dll!std::panicking::rust_panic() Line 847   Rust
    nerdbank_zcash_rust.dll!std::panicking::rust_panic_with_hook() Line 817 Rust
    nerdbank_zcash_rust.dll!std::panicking::begin_panic_handler::closure$0() Line 665   Rust
    nerdbank_zcash_rust.dll!std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::closure_env$0,never$>() Line 171    Rust
    nerdbank_zcash_rust.dll!std::panicking::begin_panic_handler() Line 652  Rust
    nerdbank_zcash_rust.dll!core::panicking::panic_fmt() Line 72    Rust
    nerdbank_zcash_rust.dll!core::panicking::panic() Line 146   Rust
>   nerdbank_zcash_rust.dll!shardtree::prunable::impl$4::insert_subtree::closure$0<sapling_crypto::tree::Node>(shardtree::prunable::impl$4::insert_subtree::closure_env$0<sapling_crypto::tree::Node>, tuple$<shardtree::tree::Tree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>,alloc::vec::Vec<shardtree::prunable::IncompleteAt,alloc::alloc::Global>>) Line 751  Rust
    nerdbank_zcash_rust.dll!enum2$<core::result::Result<tuple$<shardtree::tree::Tree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>,alloc::vec::Vec<shardtree::prunable::IncompleteAt,alloc::alloc::Global>>,enum2$<shardtree::error::InsertionError>>>::map<tuple$<shardtree::tree::Tree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>,alloc::vec::Vec<shardtree::prunable::IncompleteAt,alloc::alloc::Global>>,enum2$<shardtree::error::InsertionError>,tuple$<shardtree::tree::LocatedTree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>,alloc::vec::Vec<shardtree::prunable::IncompleteAt,alloc::alloc::Global>>,shardtree::prunable::impl$4::insert_subtree::closure_env$0<sapling_crypto::tree::Node>>(enum2$<core::result::Result<tuple$<shardtree::tree::Tree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>,alloc::vec::Vec<shardtree::prunable::IncompleteAt,alloc::alloc::Global>>,enum2$<shardtree::error::InsertionError>>> self, shardtree::prunable::impl$4::insert_subtree::closure_env$0<sapling_crypto::tree::Node>) Line 746   Rust
    nerdbank_zcash_rust.dll!shardtree::tree::LocatedTree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>::insert_subtree<sapling_crypto::tree::Node>(shardtree::tree::LocatedTree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>> self, bool subtree) Line 746  Rust
    nerdbank_zcash_rust.dll!shardtree::tree::LocatedTree<enum2$<core::option::Option<alloc::sync::Arc<sapling_crypto::tree::Node,alloc::alloc::Global>>>,tuple$<sapling_crypto::tree::Node,shardtree::prunable::RetentionFlags>>::insert_frontier_nodes<sapling_crypto::tree::Node,zcash_protocol::consensus::BlockHeight>(incrementalmerkletree::frontier::NonEmptyFrontier<sapling_crypto::tree::Node> self, enum2$<incrementalmerkletree::Retention<zcash_protocol::consensus::BlockHeight>> * frontier) Line 890    Rust
    nerdbank_zcash_rust.dll!shardtree::ShardTree<zcash_client_sqlite::wallet::commitment_tree::SqliteShardStore<ref$<rusqlite::transaction::Transaction>,sapling_crypto::tree::Node,16>,32,16>::insert_frontier_nodes<sapling_crypto::tree::Node,zcash_protocol::consensus::BlockHeight,zcash_client_sqlite::wallet::commitment_tree::SqliteShardStore<ref$<rusqlite::transaction::Transaction>,sapling_crypto::tree::Node,16>,32,16>(incrementalmerkletree::frontier::NonEmptyFrontier<sapling_crypto::tree::Node> self, enum2$<incrementalmerkletree::Retention<zcash_protocol::consensus::BlockHeight>> frontier) Line 328   Rust
    nerdbank_zcash_rust.dll!shardtree::ShardTree<zcash_client_sqlite::wallet::commitment_tree::SqliteShardStore<ref$<rusqlite::transaction::Transaction>,sapling_crypto::tree::Node,16>,32,16>::insert_frontier<sapling_crypto::tree::Node,zcash_protocol::consensus::BlockHeight,zcash_client_sqlite::wallet::commitment_tree::SqliteShardStore<ref$<rusqlite::transaction::Transaction>,sapling_crypto::tree::Node,16>,32,16>(incrementalmerkletree::frontier::Frontier<sapling_crypto::tree::Node,32> self, enum2$<incrementalmerkletree::Retention<zcash_protocol::consensus::BlockHeight>> frontier) Line 283  Rust
    nerdbank_zcash_rust.dll!zcash_client_sqlite::impl$8::put_blocks::closure$0::closure$14<zcash_protocol::consensus::Network>(zcash_client_sqlite::impl$8::put_blocks::closure$0::closure_env$14<zcash_protocol::consensus::Network> * sapling_tree, shardtree::ShardTree<zcash_client_sqlite::wallet::commitment_tree::SqliteShardStore<ref$<rusqlite::transaction::Transaction>,sapling_crypto::tree::Node,16>,32,16> *) Line 1015   Rust
    nerdbank_zcash_rust.dll!zcash_client_sqlite::impl$10::with_sapling_tree_mut<zcash_protocol::consensus::Network,zcash_client_sqlite::impl$8::put_blocks::closure$0::closure_env$14<zcash_protocol::consensus::Network>,tuple$<>,enum2$<zcash_client_sqlite::error::SqliteClientError>>(zcash_client_sqlite::WalletDb<zcash_client_sqlite::SqlTransaction,zcash_protocol::consensus::Network> * self, zcash_client_sqlite::impl$8::put_blocks::closure$0::closure_env$14<zcash_protocol::consensus::Network> callback) Line 1642  Rust
    nerdbank_zcash_rust.dll!zcash_client_sqlite::impl$8::put_blocks::closure$0<zcash_protocol::consensus::Network>(zcash_client_sqlite::impl$8::put_blocks::closure_env$0<zcash_protocol::consensus::Network> wdb, zcash_client_sqlite::WalletDb<zcash_client_sqlite::SqlTransaction,zcash_protocol::consensus::Network> *) Line 1009   Rust
    nerdbank_zcash_rust.dll!zcash_client_sqlite::WalletDb<rusqlite::Connection,zcash_protocol::consensus::Network>::transactionally<zcash_protocol::consensus::Network,zcash_client_sqlite::impl$8::put_blocks::closure_env$0<zcash_protocol::consensus::Network>,tuple$<>,enum2$<zcash_client_sqlite::error::SqliteClientError>>(zcash_client_sqlite::impl$8::put_blocks::closure_env$0<zcash_protocol::consensus::Network> self) Line 213 Rust
    nerdbank_zcash_rust.dll!zcash_client_sqlite::impl$8::put_blocks<zcash_protocol::consensus::Network>(zcash_client_sqlite::WalletDb<rusqlite::Connection,zcash_protocol::consensus::Network> * self, zcash_client_backend::data_api::chain::ChainState * from_state, alloc::vec::Vec<zcash_client_backend::data_api::ScannedBlock<zcash_client_sqlite::AccountId>,alloc::alloc::Global> blocks) Line 718  Rust
    nerdbank_zcash_rust.dll!zcash_client_backend::data_api::chain::scan_cached_blocks<zcash_protocol::consensus::Network,zcash_client_sqlite::WalletDb<rusqlite::Connection,zcash_protocol::consensus::Network>,nerdbank_zcash_rust::block_source::BlockCache>(zcash_protocol::consensus::Network * params, nerdbank_zcash_rust::block_source::BlockCache * block_source, zcash_client_sqlite::WalletDb<rusqlite::Connection,zcash_protocol::consensus::Network> * data_db, zcash_protocol::consensus::BlockHeight from_height, zcash_client_backend::data_api::chain::ChainState * from_state, unsigned __int64 limit) Line 695    Rust
    nerdbank_zcash_rust.dll!nerdbank_zcash_rust::sync::scan_blocks(zcash_protocol::consensus::Network * network, nerdbank_zcash_rust::backing_store::Db * db, zcash_client_backend::data_api::scanning::ScanRange * scan_range, zcash_client_backend::data_api::chain::ChainState * chain_state) Line 883   Rust
    nerdbank_zcash_rust.dll!nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block$1(core::pin::Pin<ref_mut$<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>>>, core::task::wake::Context *) Line 752  Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::core::impl$6::poll::closure$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(tokio::runtime::task::core::impl$6::poll::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>> ptr, enum2$<tokio::runtime::task::core::Stage<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>>> *) Line 328    Rust
    [Inline Frame] nerdbank_zcash_rust.dll!tokio::loom::std::unsafe_cell::UnsafeCell<enum2$<tokio::runtime::task::core::Stage<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>>>>::with_mut(tokio::runtime::task::core::impl$6::poll::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>> self) Line 16  Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::core::Core<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>::poll<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(core::task::wake::Context self) Line 317   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::poll_future::closure$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>) Line 485 Rust
    nerdbank_zcash_rust.dll!core::panic::unwind_safe::impl$25::call_once<enum2$<core::task::poll::Poll<enum2$<core::result::Result<nerdbank_zcash_rust::sync::DownloadAndScanResult,enum2$<nerdbank_zcash_rust::error::Error>>>>>,tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>>(core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>> self) Line 272 Rust
    nerdbank_zcash_rust.dll!std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>>,enum2$<core::task::poll::Poll<enum2$<core::result::Result<nerdbank_zcash_rust::sync::DownloadAndScanResult,enum2$<nerdbank_zcash_rust::error::Error>>>>>>(unsigned char * data) Line 559  Rust
    nerdbank_zcash_rust.dll!_ZN3std9panicking3try17hff5d2700e56173f6E() Rust
    nerdbank_zcash_rust.dll!std::panicking::try<enum2$<core::task::poll::Poll<enum2$<core::result::Result<nerdbank_zcash_rust::sync::DownloadAndScanResult,enum2$<nerdbank_zcash_rust::error::Error>>>>>,core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>>>(core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>> f) Line 523 Rust
    nerdbank_zcash_rust.dll!std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>>,enum2$<core::task::poll::Poll<enum2$<core::result::Result<nerdbank_zcash_rust::sync::DownloadAndScanResult,enum2$<nerdbank_zcash_rust::error::Error>>>>>>(core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>> f) Line 149    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::poll_future<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(tokio::runtime::task::core::Core<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>> * core, core::task::wake::Context cx) Line 491    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::Harness<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>::poll_inner<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>() Line 208 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::Harness<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>::poll<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(tokio::runtime::task::harness::Harness<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>> self) Line 153 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::raw::poll<enum2$<nerdbank_zcash_rust::sync::download_and_scan_blocks::async_fn$0::async_block_env$1>,alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(core::ptr::non_null::NonNull<tokio::runtime::task::core::Header> ptr) Line 272   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::raw::RawTask::poll(tokio::runtime::task::raw::RawTask self) Line 202  Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>::run<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>>(tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>> self) Line 417 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::impl$1::run_task::closure$0(tokio::runtime::scheduler::multi_thread::worker::impl$1::run_task::closure_env$0) Line 577 Rust
    [Inline Frame] nerdbank_zcash_rust.dll!tokio::runtime::coop::with_budget(tokio::runtime::coop::Budget) Line 107 Rust
    [Inline Frame] nerdbank_zcash_rust.dll!tokio::runtime::coop::budget(tokio::runtime::scheduler::multi_thread::worker::impl$1::run_task::closure_env$0) Line 73   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::Context::run_task(tokio::runtime::task::Notified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle,alloc::alloc::Global>> self, tokio::runtime::scheduler::multi_thread::worker::Core * task) Line 575  Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::Context::run(tokio::runtime::scheduler::multi_thread::worker::Core * self) Line 538    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure$0(tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0) Line 491 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::context::scoped::Scoped<enum2$<tokio::runtime::scheduler::Context>>::set<enum2$<tokio::runtime::scheduler::Context>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0,tuple$<>>(enum2$<tokio::runtime::scheduler::Context> * self, tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0 t) Line 40   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::context::set_scheduler::closure$0<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0>(tokio::runtime::context::set_scheduler::closure_env$0<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0> c, tokio::runtime::context::Context *) Line 176   Rust
    nerdbank_zcash_rust.dll!std::thread::local::LocalKey<tokio::runtime::context::Context>::try_with<tokio::runtime::context::Context,tokio::runtime::context::set_scheduler::closure_env$0<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0>,tuple$<>>(tokio::runtime::context::set_scheduler::closure_env$0<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0> self) Line 286    Rust
    nerdbank_zcash_rust.dll!std::thread::local::LocalKey<tokio::runtime::context::Context>::with<tokio::runtime::context::Context,tokio::runtime::context::set_scheduler::closure_env$0<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0>,tuple$<>>(tokio::runtime::context::set_scheduler::closure_env$0<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0> self) Line 262    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::context::set_scheduler<tuple$<>,tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0>(enum2$<tokio::runtime::scheduler::Context> * v, tokio::runtime::scheduler::multi_thread::worker::run::closure$0::closure_env$0) Line 177    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::run::closure$0(tokio::runtime::scheduler::multi_thread::worker::run::closure_env$0, tokio::runtime::context::blocking::BlockingRegionGuard *) Line 486 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::worker::run::closure_env$0,tuple$<>>(enum2$<tokio::runtime::scheduler::Handle> * handle, bool allow_block_in_place, tokio::runtime::scheduler::multi_thread::worker::run::closure_env$0) Line 65   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::run(alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::worker::Worker,alloc::alloc::Global> worker) Line 478    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure$0(tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0) Line 447 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::blocking::task::impl$2::poll<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0,tuple$<>>(core::pin::Pin<ref_mut$<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>>> self, core::task::wake::Context * _cx) Line 42 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::core::impl$6::poll::closure$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>(tokio::runtime::task::core::impl$6::poll::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule> ptr, enum2$<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>>> *) Line 328  Rust
    [Inline Frame] nerdbank_zcash_rust.dll!tokio::loom::std::unsafe_cell::UnsafeCell<enum2$<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>>>>::with_mut(tokio::runtime::task::core::impl$6::poll::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule> self) Line 16    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>(core::task::wake::Context self) Line 317   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::poll_future::closure$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>(tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>) Line 485 Rust
    nerdbank_zcash_rust.dll!core::panic::unwind_safe::impl$25::call_once<enum2$<core::task::poll::Poll<tuple$<>>>,tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>>(core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>> self) Line 273 Rust
    nerdbank_zcash_rust.dll!std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>>,enum2$<core::task::poll::Poll<tuple$<>>>>(unsigned char * data) Line 559  Rust
    nerdbank_zcash_rust.dll!_ZN3std9panicking3try8do_catch17he922420012114a9aE()    Rust
    nerdbank_zcash_rust.dll!std::panicking::try<enum2$<core::task::poll::Poll<tuple$<>>>,core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>>>(core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>> f) Line 523 Rust
    nerdbank_zcash_rust.dll!std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>>,enum2$<core::task::poll::Poll<tuple$<>>>>(core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::closure_env$0<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>> f) Line 149    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::poll_future<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>(tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule> * core, core::task::wake::Context cx) Line 491    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>::poll_inner<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>() Line 208 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>(tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule> self) Line 153 Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::raw::poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::impl$0::launch::closure_env$0>,tokio::runtime::blocking::schedule::BlockingSchedule>(core::ptr::non_null::NonNull<tokio::runtime::task::core::Header> ptr) Line 272   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::raw::RawTask::poll(tokio::runtime::task::raw::RawTask self) Line 202  Rust
    nerdbank_zcash_rust.dll!tokio::runtime::task::UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule>::run<tokio::runtime::blocking::schedule::BlockingSchedule>(tokio::runtime::task::UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule> self) Line 453   Rust
    nerdbank_zcash_rust.dll!tokio::runtime::blocking::pool::Task::run(tokio::runtime::blocking::pool::Task) Line 160    Rust
    nerdbank_zcash_rust.dll!tokio::runtime::blocking::pool::Inner::run(unsigned __int64 self) Line 513  Rust
    nerdbank_zcash_rust.dll!tokio::runtime::blocking::pool::impl$6::spawn_thread::closure$0(tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0) Line 471   Rust
    nerdbank_zcash_rust.dll!std::sys_common::backtrace::__rust_begin_short_backtrace<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>(tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0 f) Line 161  Rust
    nerdbank_zcash_rust.dll!std::thread::impl$0::spawn_unchecked_::closure$2::closure$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>(std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>) Line 543   Rust
    nerdbank_zcash_rust.dll!core::panic::unwind_safe::impl$25::call_once<tuple$<>,std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>>(core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>> self) Line 273   Rust
    nerdbank_zcash_rust.dll!std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>>,tuple$<>>(unsigned char * data) Line 561 Rust
    nerdbank_zcash_rust.dll!_ZN3std9panicking3try8do_catch17he922420012114a9aE()    Rust
    nerdbank_zcash_rust.dll!std::panicking::try<tuple$<>,core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>>>(core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>> f) Line 523   Rust
    [Inline Frame] nerdbank_zcash_rust.dll!std::panic::catch_unwind(core::panic::unwind_safe::AssertUnwindSafe<std::thread::impl$0::spawn_unchecked_::closure$2::closure_env$0<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>> f) Line 149   Rust
    nerdbank_zcash_rust.dll!std::thread::impl$0::spawn_unchecked_::closure$2<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>(std::thread::impl$0::spawn_unchecked_::closure_env$2<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>) Line 541 Rust
    nerdbank_zcash_rust.dll!core::ops::function::FnOnce::call_once<std::thread::impl$0::spawn_unchecked_::closure_env$2<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>>,tuple$<>>(std::thread::impl$0::spawn_unchecked_::closure_env$2<tokio::runtime::blocking::pool::impl$6::spawn_thread::closure_env$0,tuple$<>> *) Line 250  Rust
    [Inline Frame] nerdbank_zcash_rust.dll!alloc::boxed::impl$48::call_once() Line 2022 Rust
    [Inline Frame] nerdbank_zcash_rust.dll!alloc::boxed::impl$48::call_once() Line 2022 Rust
    nerdbank_zcash_rust.dll!std::sys::pal::windows::thread::impl$0::new::thread_start() Line 52 Rust
    kernel32.dll!00007fff7c85257d() Unknown
    ntdll.dll!00007fff7d02af28()    Unknown

[nuttycom]

@AArnott thanks very much for the report. Is there any chance that you can provide the contents of the scan queue (along with the current range being scanned) at the time that you encounter this error?

[AArnott]

scan range at time of panic: 2516985..2523970

block_range_start	block_range_end	priority
419200	2224314	0
2224314	2516985	20
2516985	2546643	50

[nuttycom]

https://github.com/zcash/incrementalmerkletree/pull/107 removes the assertion that is at fault here, and improves the distinction between pruned nodes and inserted nodes. A librustzcash PR will follow from this, but the shardtree change should be sufficient to work around your problem. @AArnott can you try it out?

[AArnott]

Yes, I can confirm the panic goes away with that shardtree change.

[nuttycom]

The summary here is that the assertion that was added in conjunction with the previous bugfix was testing a flaky condition that doesn't work correctly with how subtree roots were previously being inserted. #1439 will change librustzcash to change how it does subtree root insertion to add a semantic distinction between fully-scanned-and-pruned subtree roots and inserted subtree roots, but I don't think it will be a good idea to reintroduce the assertion irrespective of this change.