Runtime panic with zombienet privacy sidechain setup

[clangenb]

Zombienet version: 1.3.98 Latest master at https://github.com/integritee-network/parachain/commit/a57436ba3771b2d1936cfb1776a28c9f2db581ad

Version: 1.9.4-a57436ba377

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/alloc/src/boxed.rs:2021:9
      std::panicking::rust_panic_with_hook
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/panicking.rs:783:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/panicking.rs:657:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/sys_common/backtrace.rs:170:18
   4: rust_begin_unwind
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/panicking.rs:645:5
   5: core::panicking::panic_fmt
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/core/src/panicking.rs:72:14
   6: core::panicking::assert_failed_inner
   7: core::panicking::assert_failed
   8: cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::enqueue_inbound_downward_messages
   9: frame_support::storage::transactional::with_transaction
  10: <cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}
  11: environmental::using_once
  12: <integritee_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter
  13: <integritee_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch
  14: <sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply
  15: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic
  16: integritee_runtime::api::dispatch
  17: sp_externalities::scope_limited::ext::using
  18: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
  19: sc_executor::wasm_runtime::RuntimeCache::with_instance
  20: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  21: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
  22: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  23: <integritee_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_offchain::OffchainWorkerApi<__SrApiBlock__>>::__runtime_api_internal_call_api_at
  24: sp_block_builder::BlockBuilder::apply_extrinsic
  25: <integritee_runtime::RuntimeApiImpl<Block,C> as sp_api::ApiExt<Block>>::execute_in_transaction
  26: <sc_basic_authorship::basic_authorship::Proposer<Block,C,A,PR> as sp_consensus::Proposer<Block>>::propose::{{closure}}
  27: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  28: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  29: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  30: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  31: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  32: tokio::runtime::task::core::Core<T,S>::poll
  33: tokio::runtime::task::harness::Harness<T,S>::poll
  34: std::sys_common::backtrace::__rust_begin_short_backtrace
  35: core::ops::function::FnOnce::call_once{{vtable.shim}}
  36: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/alloc/src/boxed.rs:2007:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/alloc/src/boxed.rs:2007:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/sys/unix/thread.rs:108:17
  37: start_thread
             at ./nptl/pthread_create.c:442:8
  38: __GI___clone3
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 'tokio-runtime-worker' panicked at 'assertion `left == right` failed
  left: 0x0000000000000000000000000000000000000000000000000000000000000000
 right: 0x3f613ca13c4f8f0d321683e747251fd63c9df3681a22cdda1762bdbe85fbb910', /home/clang/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.7.0/src/lib.rs:1164

This is a bug. Please report it at:

        https://github.com/integritee-network/parachain/issues/new

====================

Version: 1.9.4-a57436ba377

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/alloc/src/boxed.rs:2021:9
      std::panicking::rust_panic_with_hook
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/panicking.rs:783:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/panicking.rs:657:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/sys_common/backtrace.rs:170:18
   4: rust_begin_unwind
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/panicking.rs:645:5
   5: core::panicking::panic_fmt
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/core/src/panicking.rs:72:14
   6: core::panicking::panic_display
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/core/src/panicking.rs:178:5
      core::panicking::panic_str
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/core/src/panicking.rs:152:5
      core::option::expect_failed
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/core/src/option.rs:1979:5
   7: <(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize
   8: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook
   9: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block
  10: integritee_runtime::api::dispatch
  11: sp_externalities::scope_limited::ext::using
  12: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
  13: sc_executor::wasm_runtime::RuntimeCache::with_instance
  14: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  15: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
  16: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  17: <integritee_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_offchain::OffchainWorkerApi<__SrApiBlock__>>::__runtime_api_internal_call_api_at
  18: sp_block_builder::BlockBuilder::finalize_block
  19: sc_block_builder::BlockBuilder<Block,C>::build
  20: <sc_basic_authorship::basic_authorship::Proposer<Block,C,A,PR> as sp_consensus::Proposer<Block>>::propose::{{closure}}
  21: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  22: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  23: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  24: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  25: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  26: tokio::runtime::task::core::Core<T,S>::poll
  27: tokio::runtime::task::harness::Harness<T,S>::poll
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
  29: core::ops::function::FnOnce::call_once{{vtable.shim}}
  30: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/alloc/src/boxed.rs:2007:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/alloc/src/boxed.rs:2007:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/9d83ac217957eece2189eccf4a7232caec7232ee/library/std/src/sys/unix/thread.rs:108:17
  31: start_thread
             at ./nptl/pthread_create.c:442:8
  32: __GI___clone3
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 'tokio-runtime-worker' panicked at 'set_validation_data inherent needs to be present in every block!', /home/clang/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.7.0/src/lib.rs:265

This is a bug. Please report it at:

        https://github.com/integritee-network/parachain/issues/new

[brenzi]

This looks like the error we saw on paseo whenever we force a new Head (regenesis) Alejandro fixed that with a setStorage call on relay taking the expected value from this log. I believe this is not a runtime panic. It's a node issue

[clangenb]

I agree that it is somehow a node issue, as the runtime is expecting an inherent that is not present. Also, it is not consistent, block production works, but every once in a while, the runtime panics because of this missing inherent and that block fails to be produced then.

[clangenb]

Actually no, the parachain doesn't produce blocks...

[brenzi]

Can you try the paseo with integritee zombie config plz? Worked for me with pdot 1.9.0 relay

[clangenb]

Something was wrong with the relay chain binaries. I redownloaded them, and it started to work.