Storage root not match after forkless upgraded

badkk commented 4 years ago
Context

Repo: Crust
Steps:
Start maxwell network
Arrange an upgrade, change spec_version from 1 to 2
Connect new node to network with old old binary
Report that storage root not match, but can still syncing block
Version

rc4
Error log

➜  crust git:(release/0.7.0) ./target/release/crust --base-path /tmp/maxwell --chain maxwell --bootnodes /ip4/106.75.63.71/tcp/30333/p2p/12D3KooWLvTTA2SGZ5XkhFBn3pzkk7cNWv2BL3YcUygdA5GpLFcb
2020-08-14 14:58:35 Crust Node
2020-08-14 14:58:35 ✌️  version 0.7.0-3778e05-x86_64-macos
2020-08-14 14:58:35 ❤️  by crustio, 2019-2020
2020-08-14 14:58:35 📋 Chain specification: Maxwell
2020-08-14 14:58:35 🏷  Node name: frantic-treatment-0897
2020-08-14 14:58:35 👤 Role: FULL
2020-08-14 14:58:35 💾 Database: RocksDb at /tmp/maxwell/chains/maxwell1/db
2020-08-14 14:58:35 ⛓  Native runtime: crust-1 (crustio-crust-1.tx1.au1)
2020-08-14 14:58:35 🔨 Initializing Genesis block/state (state: 0x1d4a…08c0, header-hash: 0xa5be…dd30)
2020-08-14 14:58:35 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2020-08-14 14:58:35 ⏱  Loaded block-time = 6000 milliseconds from genesis on first-launch
2020-08-14 14:58:35 👶 Creating empty BABE epoch changes on what appears to be first startup.
2020-08-14 14:58:35 📦 Highest known block at #0
2020-08-14 14:58:35 🏷  Local node identity is: 12D3KooWMd2LghHrKLWHNmwXSxybBUzemX1rttQfzkWYnEWGnpUe (legacy representation: QmTFpx114ZgjvVA2UXZaJdFxit4QFQWL6fzvgAKxt3Uijp)
2020-08-14 14:58:35 〽️ Prometheus server started at 127.0.0.1:9615
2020-08-14 14:58:35 🔍 Discovered new external address for our node: /ip4/27.115.27.218/tcp/30333/p2p/12D3KooWMd2LghHrKLWHNmwXSxybBUzemX1rttQfzkWYnEWGnpUe
2020-08-14 14:58:35 🔍 Discovered new external address for our node: /ip4/192.168.50.1/tcp/30333/p2p/12D3KooWMd2LghHrKLWHNmwXSxybBUzemX1rttQfzkWYnEWGnpUe
2020-08-14 14:58:35 🔍 Discovered new external address for our node: /ip4/127.0.0.1/tcp/30333/p2p/12D3KooWMd2LghHrKLWHNmwXSxybBUzemX1rttQfzkWYnEWGnpUe
Hash: given=fe046d57776e24303d43357e7a1727ad0e99302580fab5531afaeef9f5ebb17a, expected=9a6af4344d5d0ab40033f642e1ff23c61c8e6e458c3d91c87b434300ec5d297f

====================

Version: 0.7.0-3778e05-x86_64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic
   5: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllModules,COnRuntimeUpgrade>::execute_block
   6: <crust_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,crust_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<crust_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<crust_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<crust_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<crust_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<crust_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<crust_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<crust_runtime::Runtime>)>>>>::execute_block
   7: sp_api::runtime_decl_for_Core::execute_block_native_call_generator::{{closure}}
   8: std::thread::local::LocalKey<T>::with
   9: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  10: sc_executor::wasm_runtime::RuntimeCache::with_instance
  11: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  12: sp_state_machine::StateMachine<B,H,N,Exec>::execute_aux
  13: sp_state_machine::StateMachine<B,H,N,Exec>::execute_using_consensus_failure_handler
  14: <sc_service::client::call_executor::LocalCallExecutor<B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  15: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  16: sp_api::runtime_decl_for_Core::execute_block_call_api_at
  17: sp_api::Core::execute_block_with_context
  18: <&sc_service::client::client::Client<B,E,Block,RA> as sp_consensus::block_import::BlockImport<Block>>::import_block
  19: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sp_consensus::block_import::BlockImport<Block>>::import_block
  20: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sp_consensus::block_import::BlockImport<Block>>::import_block
  21: sp_consensus::import_queue::import_single_block_metered
  22: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  23: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  24: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
  25: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  26: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  27: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  29: std::thread::local::LocalKey<T>::with
  30: futures_executor::local_pool::block_on
  31: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  32: tokio::runtime::task::core::Core<T,S>::poll
  33: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  34: tokio::runtime::task::harness::Harness<T,S>::poll
  35: tokio::runtime::blocking::pool::Inner::run
  36: tokio::runtime::context::enter
  37: std::sys_common::backtrace::__rust_begin_short_backtrace
  38: core::ops::function::FnOnce::call_once{{vtable.shim}}
  39: std::sys::unix::thread::Thread::new::thread_start
  40: __pthread_start

Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /Users/kun/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13

This is a bug. Please report it at:

    https://github.com/crustio/crust/issues/new

Hash: given=85148d96cc33f1d4f966e2531ac3d4c410cacc2a2d5570150eb9c4e784f3a401, expected=c50adb50b02de8729a5a10a05c919c99d3e4326412cbd063edbb92c0101ecb3a

====================

Version: 0.7.0-3778e05-x86_64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic
   5: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllModules,COnRuntimeUpgrade>::execute_block
   6: <crust_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,crust_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<crust_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<crust_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<crust_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<crust_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<crust_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<crust_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<crust_runtime::Runtime>)>>>>::execute_block
   7: sp_api::runtime_decl_for_Core::execute_block_native_call_generator::{{closure}}
   8: std::thread::local::LocalKey<T>::with
   9: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  10: sc_executor::wasm_runtime::RuntimeCache::with_instance
  11: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  12: sp_state_machine::StateMachine<B,H,N,Exec>::execute_aux
  13: sp_state_machine::StateMachine<B,H,N,Exec>::execute_using_consensus_failure_handler
  14: <sc_service::client::call_executor::LocalCallExecutor<B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  15: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  16: sp_api::runtime_decl_for_Core::execute_block_call_api_at
  17: sp_api::Core::execute_block_with_context
  18: <&sc_service::client::client::Client<B,E,Block,RA> as sp_consensus::block_import::BlockImport<Block>>::import_block
  19: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sp_consensus::block_import::BlockImport<Block>>::import_block
  20: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sp_consensus::block_import::BlockImport<Block>>::import_block
  21: sp_consensus::import_queue::import_single_block_metered
  22: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  23: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  24: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
  25: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  26: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  27: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  29: std::thread::local::LocalKey<T>::with
  30: futures_executor::local_pool::block_on
  31: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  32: tokio::runtime::task::core::Core<T,S>::poll
  33: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  34: tokio::runtime::task::harness::Harness<T,S>::poll
  35: tokio::runtime::blocking::pool::Inner::run
  36: tokio::runtime::context::enter
  37: std::sys_common::backtrace::__rust_begin_short_backtrace
  38: core::ops::function::FnOnce::call_once{{vtable.shim}}
  39: std::sys::unix::thread::Thread::new::thread_start
  40: __pthread_start

Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /Users/kun/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13

This is a bug. Please report it at:

    https://github.com/crustio/crust/issues/new

Hash: given=96d980ccd3f875eeebed9ac1c50566a38e75db198cf11c2b47ded544e18dec66, expected=8be5d9bb2906012d41dc7970deecc0b85cf731d24dde0e3b779b6be6e521abb5

====================

Version: 0.7.0-3778e05-x86_64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic
   5: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllModules,COnRuntimeUpgrade>::execute_block
   6: <crust_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,crust_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<crust_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<crust_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<crust_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<crust_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<crust_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<crust_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<crust_runtime::Runtime>)>>>>::execute_block
   7: sp_api::runtime_decl_for_Core::execute_block_native_call_generator::{{closure}}
   8: std::thread::local::LocalKey<T>::with
   9: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  10: sc_executor::wasm_runtime::RuntimeCache::with_instance
  11: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  12: sp_state_machine::StateMachine<B,H,N,Exec>::execute_aux
  13: sp_state_machine::StateMachine<B,H,N,Exec>::execute_using_consensus_failure_handler
  14: <sc_service::client::call_executor::LocalCallExecutor<B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  15: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  16: sp_api::runtime_decl_for_Core::execute_block_call_api_at
  17: sp_api::Core::execute_block_with_context
  18: <&sc_service::client::client::Client<B,E,Block,RA> as sp_consensus::block_import::BlockImport<Block>>::import_block
  19: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sp_consensus::block_import::BlockImport<Block>>::import_block
  20: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sp_consensus::block_import::BlockImport<Block>>::import_block
  21: sp_consensus::import_queue::import_single_block_metered
  22: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  23: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  24: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
  25: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  26: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  27: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  29: std::thread::local::LocalKey<T>::with
  30: futures_executor::local_pool::block_on
  31: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  32: tokio::runtime::task::core::Core<T,S>::poll
  33: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  34: tokio::runtime::task::harness::Harness<T,S>::poll
  35: tokio::runtime::blocking::pool::Inner::run
  36: tokio::runtime::context::enter
  37: std::sys_common::backtrace::__rust_begin_short_backtrace
  38: core::ops::function::FnOnce::call_once{{vtable.shim}}
  39: std::sys::unix::thread::Thread::new::thread_start
  40: __pthread_start

Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /Users/kun/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13

This is a bug. Please report it at:

    https://github.com/crustio/crust/issues/new

Hash: given=1fb606cbe8cf369d3ff130647d53ff61f6a677d0288b6b2c1ac6fb9ed87dc3cc, expected=f7e930bcbf0380e9c1c30b8125e471f2756680b4b37d7f9e94798c144e7821ab

====================

Version: 0.7.0-3778e05-x86_64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic
   5: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllModules,COnRuntimeUpgrade>::execute_block
   6: <crust_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,crust_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<crust_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<crust_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<crust_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<crust_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<crust_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<crust_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<crust_runtime::Runtime>)>>>>::execute_block
   7: sp_api::runtime_decl_for_Core::execute_block_native_call_generator::{{closure}}
   8: std::thread::local::LocalKey<T>::with
   9: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  10: sc_executor::wasm_runtime::RuntimeCache::with_instance
  11: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  12: sp_state_machine::StateMachine<B,H,N,Exec>::execute_aux
  13: sp_state_machine::StateMachine<B,H,N,Exec>::execute_using_consensus_failure_handler
  14: <sc_service::client::call_executor::LocalCallExecutor<B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  15: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  16: sp_api::runtime_decl_for_Core::execute_block_call_api_at
  17: sp_api::Core::execute_block_with_context
  18: <&sc_service::client::client::Client<B,E,Block,RA> as sp_consensus::block_import::BlockImport<Block>>::import_block
  19: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sp_consensus::block_import::BlockImport<Block>>::import_block
  20: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sp_consensus::block_import::BlockImport<Block>>::import_block
  21: sp_consensus::import_queue::import_single_block_metered
  22: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  23: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  24: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
  25: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  26: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  27: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  29: std::thread::local::LocalKey<T>::with
  30: futures_executor::local_pool::block_on
  31: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  32: tokio::runtime::task::core::Core<T,S>::poll
  33: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  34: tokio::runtime::task::harness::Harness<T,S>::poll
  35: tokio::runtime::blocking::pool::Inner::run
  36: tokio::runtime::context::enter
  37: std::sys_common::backtrace::__rust_begin_short_backtrace
  38: core::ops::function::FnOnce::call_once{{vtable.shim}}
  39: std::sys::unix::thread::Thread::new::thread_start
  40: __pthread_start

Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /Users/kun/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd/macros.rs:13

This is a bug. Please report it at:

    https://github.com/crustio/crust/issues/new

2020-08-14 14:58:40 ⚙️  Syncing, target=#244437 (6 peers), best: #3338 (0x4753…eb49), finalized #3101 (0xa5b7…023a), ⬇ 516.7kiB/s ⬆ 5.5kiB/s
bkchr commented 4 years ago
So you just changed the spec_version?
Is that a local test net?
badkk commented 4 years ago
So you just changed the spec_version?

Is that a local test net?
And with runtime change.
bkchr commented 4 years ago
How can we reproduce this?
badkk commented 4 years ago
How can we reproduce this?
Binary mode

Clone Crust repo and git checkout release/0.7.0
Run with maxwell ./target/release/crust --chain maxwell --base-path /tmp/maxwell
Docker mode

docker run crustio/crust:0.7.0 ./crust --base-path /tmp/maxwell --chain maxwell
bkchr commented 4 years ago
I used the docker image and could not reproduce it
badkk commented 4 years ago
I used the docker image and could not reproduce it
Pretty strange, what is your os? Mine was OSX and can 100% reproduce this error(both docker and binary) at the beginning of syncing block. So is this possibly a bug under OSX?
bkchr commented 4 years ago
I'm using Linux.
mmyyrroonn commented 3 years ago
@zikunfan Could you share the status of this issue?
paritytech / substrate
