Indexing breaks at block #22633806

[frol]

May 21 03:46:47.817 INFO indexer_for_explorer: Block height 22633799
May 21 03:46:47.848 INFO indexer_for_explorer: Block height 22633800
May 21 03:46:47.862 INFO indexer_for_explorer: Block height 22633801
May 21 03:46:47.880 INFO indexer_for_explorer: Block height 22633802
May 21 03:46:47.893 INFO indexer_for_explorer: Block height 22633803
May 21 03:46:47.904 INFO indexer_for_explorer: Block height 22633804
May 21 03:46:47.953 INFO indexer_for_explorer: Block height 22633805
May 21 03:46:47.964 INFO indexer_for_explorer: Block height 22633806
thread 'main' panicked at '`at` split index (is 1) should be <= len (is 0)', library/alloc/src/vec/mod.rs:1697:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/core/src/panicking.rs:92:14
   2: alloc::vec::Vec<T,A>::split_off::assert_failed
             at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/alloc/src/vec/mod.rs:1697:13
   3: near_indexer::streamer::build_streamer_message::{{closure}}
   4: near_indexer::streamer::start::{{closure}}
   5: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
   6: tokio::runtime::task::harness::Harness<T,S>::poll
   7: std::thread::local::LocalKey<T>::with
   8: tokio::task::local::LocalSet::tick
   9: tokio::macros::scoped_tls::ScopedKey<T>::set
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::macros::scoped_tls::ScopedKey<T>::set
  13: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
  14: tokio::runtime::Runtime::block_on
  15: actix_rt::system::SystemRunner::run
  16: indexer_explorer::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
May 21 03:46:53.498 INFO stats: #37888747  Dk5tnvZpiwmUNie9AR5aVQSrvGQexDABTYkRMZyZvrZ -/59 34/29/40 peers ⬇ 280.5kiB/s ⬆ 209.2kiB/s 1.09 bps 13.31 Tgas/s CPU: 79%, Mem: 7.8 GiB

[bowenwang1996]

This looks bad. What happened here? Is it mainnet?

[khorolets]

@bowenwang1996 yes, mainnet, I'm investigating, will update you.

[frol]

@bowenwang1996 Your commit https://github.com/near/nearcore/commit/584aebe20e85b2584b5e2bcc1b7a0461cddd2486 resolved the issue. Please, land it on master

[bowenwang1996]

@frol I prefer to wait to see whether it breaks at any other block. My fix is ad-hoc as I don't understand the underlying cause.

[khorolets]

@bowenwang1996 I approve that no other problematic blocks like that were met during indexing. I guess we can land that commit on master already

[bowenwang1996]

@khorolets okay I'll take another look to see if I can find anything suspicious. If this is indeed deterministic, it means that we probably have some bug somewhere.