search

moonbeam-foundation / moonbeam

An Ethereum-compatible smart contract parachain on Polkadot
https://moonbeam.network
GNU General Public License v3.0
921 stars 331 forks source link

Storage root must match that calculated Error when Syncing Polkadot #1075

Closed luckyve closed 2 years ago

luckyve commented 2 years ago

Version: 0.16.0-17c104d-x86_64-linux-gnu 0: sp_panic_handler::set::{{closure}} 1: std::panicking::rust_panic_with_hook at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/panicking.rs:628:17 2: std::panicking::begin_panic::{{closure}} 3: std::sys_common::backtrace::rust_end_short_backtrace 4: std::panicking::begin_panic 5: <polkadot_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,()>,polkadot_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion,frame_system::extensions::check_tx_version::CheckTxVersion,frame_system::extensions::check_genesis::CheckGenesis,frame_system::extensions::check_mortality::CheckMortality,frame_system::extensions::check_nonce::CheckNonce,frame_system::extensions::check_weight::CheckWeight,pallet_transaction_payment::ChargeTransactionPayment,polkadot_runtime_common::claims::PrevalidateAttests)>>>>::execute_block 6: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}} 7: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux 8: <polkadot_runtime::RuntimeApiImpl<SR_API_BLOCK,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_execute_block_runtime_api_impl 9: <core::future::from_generator::GenFuture as core::future::future::Future>::poll 10: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport>::import_block::{{closure}} 11: <core::future::from_generator::GenFuture as core::future::future::Future>::poll 12: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport>::import_block::{{closure}} 13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll 14: <core::future::from_generator::GenFuture as core::future::future::Future>::poll 15: <core::future::from_generator::GenFuture as core::future::future::Future>::poll 16: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll 17: <tracing_futures::Instrumented as core::future::future::Future>::poll 18: tokio::runtime::task::raw::poll 19: std::sys_common::backtrace::rust_begin_short_backtrace 20: core::ops::function::FnOnce::call_once{{vtable.shim}} 21: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/boxed.rs:1636:9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/boxed.rs:1636:9 std::sys::unix::thread::Thread::new::thread_start at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/sys/unix/thread.rs:106:17 22: start_thread 23: clone Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /home/gh-actions/.cargo/git/checkouts/substrate-189071a041b0d328/f0d3cd3/frame/executive/src/lib.rs:503 This is a bug. Please report it at:

011https://github.com/PureStake/moonbeam/issues/new

crystalin commented 2 years ago

@luckyve what command did you run ?

luckyve commented 2 years ago

@luckyve what command did you run ?

ExecStart=/home/moonbeam-znver3 --validator --port 30333 --rpc-port 9933 --ws-port 9944 --pruning=archive --execution wasm --wasm-execution compiled --state-cache-size 1 --db-cache 65536 --base-path /home/moonbeam-data --chain moonbeam --name luckyve2 --out-peers 50 --in-peers 50 \ -- \ --port 30334 --rpc-port 9934 --ws-port 9945 --pruning=archive --out-peers 50 --in-peers 50 --name luckyve2_moonbeam_relay

luckyve commented 2 years ago

now with the relaychain synchronized does not appear anymore

crystalin commented 2 years ago

Additional information for Parity (cc @bkchr):

crystalin commented 2 years ago

Another report:

Dec 11 11:51:02 moonbeam1 moonbeam[16820]: 2021-12-11 11:51:02 [🌗] 💤 Idle (0 peers), best: #0 (0xfe58…b76d), finalized #0 (0xfe58…b76d), ⬇ 0 ⬆ 0
Dec 11 11:51:04 moonbeam1 moonbeam[16820]: 2021-12-11 11:51:04 [Relaychain] ⚙️  Syncing 54.4 bps, target=#8086675 (50 peers), best: #7621056 (0x8fa8…75c9), finalized #7620609 (0x192c…c358), ⬇ 4.0kiB/s ⬆ 6.0kiB/s
Dec 11 11:51:06 moonbeam1 moonbeam[16820]: Hash: given=cff3b1b47454b8789a134862b146d74f1b76c6549e6d7c7a1352a90da1ae9356, expected=a26dbe3010523ebe88ac37c6ca7b9392336016b6fa543628fe3d2509019684d3
Dec 11 11:51:06 moonbeam1 moonbeam[16820]: ====================
Dec 11 11:51:06 moonbeam1 moonbeam[16820]: Version: 0.16.0-17c104d-x86_64-linux-gnu
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    0: sp_panic_handler::set::{{closure}}
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    1: std::panicking::rust_panic_with_hook
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:              at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/panicking.rs:628:17
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    2: std::panicking::begin_panic::{{closure}}
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    3: std::sys_common::backtrace::__rust_end_short_backtrace
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    4: std::panicking::begin_panic
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    5: <polkadot_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,()>,polkadot_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<polkadot_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<polkadot_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<polkadot_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<polkadot_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<polkadot_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<polkadot_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<polkadot_runtime::Runtime>,polkadot_runtime_common::claims::PrevalidateAttests<polkadot_runtime::Runtime>)>>>>::execute_block
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    6: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    7: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    8: <polkadot_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_execute_block_runtime_api_impl
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:    9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   10: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   12: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   16: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   17: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   18: tokio::runtime::task::raw::poll
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   19: std::sys_common::backtrace::__rust_begin_short_backtrace
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   20: core::ops::function::FnOnce::call_once{{vtable.shim}}
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   21: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:              at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/boxed.rs:1636:9
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:              at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/alloc/src/boxed.rs:1636:9
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:       std::sys::unix::thread::Thread::new::thread_start
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:              at rustc/497ee321af3b8496eaccd7af7b437f18bab81abf/library/std/src/sys/unix/thread.rs:106:17
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   22: start_thread
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:   23: clone
Dec 11 11:51:06 moonbeam1 moonbeam[16820]: Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /home/gh-actions/.cargo/git/checkouts/substrate-189071a041b0d328/f0d3cd3/frame/executive/src/lib.rs:503
Dec 11 11:51:06 moonbeam1 moonbeam[16820]: This is a bug. Please report it at:
Dec 11 11:51:06 moonbeam1 moonbeam[16820]:         https://github.com/PureStake/moonbeam/issues/new
Dec 11 11:51:07 moonbeam1 moonbeam[16820]: 2021-12-11 11:51:07 [🌗] 💤 Idle (0 peers), best: #0 (0xfe58…b76d), finalized #0 (0xfe58…b76d), ⬇ 0 ⬆ 0
Dec 11 11:51:09 moonbeam1 moonbeam[16820]: 2021-12-11 11:51:09 [Relaychain] ⚙️  Syncing 53.8 bps, target=#8086676 (50 peers), best: #7621325 (0xa0a9…3ffb), finalized #7621120 (0x000d…5ab2), ⬇ 7.6kiB/s ⬆ 9.2kiB/s
crystalin commented 2 years ago

The issue seems to be a mismatch between the runtime executed vs the one expected. This happens because:

  1. The node runs nativeElseWasm as default for the relay side of the node
  2. The relay native runtime is built using our modified polkadot/substrate which adds democracy events to the code (see https://github.com/PureStake/substrate/commit/f0d3cd391dc052daa613d997215137f84eeb77a2)

The fact the relay block failing (around best: #7621056) is happening after the runtime 9122 upgrade (#7560558) coincide with the explanation.

The solution is to run --execution wasm also for the relay side (after the -- parameter)