error log on collator before restarting on runtime 6550 upgrade

2024-08-20 03:17:00 [Parachain] 🙌 Starting consensus session on top of parent 0x3791a0253e1dbd448912bc79f38af78ffa699099c8eec346cf5232716fd12dde
2024-08-20 03:17:00 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-38f315c8db144fd3/5811785/cumulus/pallets/aura-ext/src/consensus_hook.rs:69:9:
assertion `left == right` failed: slot number mismatch
  left: Slot(143676985)
 right: Slot(287353970)
2024-08-20 03:17:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:00 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4f44e1 - <unknown>!rust_begin_unwind
    1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0xcd17 - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x247e96 - <unknown>!core::panicking::assert_failed::h1620870d51024939
    4: 0x27cc08 - <unknown>!frame_support::storage::transactional::with_transaction::h879f21777e24b6eb
    5: 0x43c821 - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h6fce7722202d860e
    6: 0x44646a - <unknown>!frame_support::dispatch_context::run_in_context::h04b55aceefdb7344
    7: 0x70b26 - <unknown>!<darwinia_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h9a64b8b9f9986198
    8: 0x66e57 - <unknown>!<darwinia_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h681bfff0c0c825b3
    9: 0x2ee377 - <unknown>!<fp_self_contained::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra,SelfContainedSignedInfo> as sp_runtime::traits::Applyable>::apply::h5000ae82748e94a6
   10: 0x3f14bc - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::haf6b3ddee5a7c9b9
   11: 0x3ac216 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.
2024-08-20 03:17:00 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-38f315c8db144fd3/5811785/substrate/frame/aura/src/lib.rs:416:9:
assertion `left == right` failed: Timestamp slot must match `CurrentSlot`
  left: Slot(143676985)
 right: Slot(287353970)
2024-08-20 03:17:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:00 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4f44e1 - <unknown>!rust_begin_unwind
    1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0xcd17 - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x247e96 - <unknown>!core::panicking::assert_failed::h1620870d51024939
    4: 0x278ea3 - <unknown>!frame_support::storage::transactional::with_transaction::h7de79e7af2ee1a2b
    5: 0x4478db - <unknown>!frame_support::dispatch_context::run_in_context::h1ed7a2444526a551
    6: 0x70b87 - <unknown>!<darwinia_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h9a64b8b9f9986198
    7: 0x66e57 - <unknown>!<darwinia_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h681bfff0c0c825b3
    8: 0x2ee377 - <unknown>!<fp_self_contained::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra,SelfContainedSignedInfo> as sp_runtime::traits::Applyable>::apply::h5000ae82748e94a6
    9: 0x3f14bc - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::haf6b3ddee5a7c9b9
   10: 0x3ac216 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.
2024-08-20 03:17:00 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-38f315c8db144fd3/5811785/cumulus/pallets/parachain-system/src/lib.rs:265:18:
set_validation_data inherent needs to be present in every block!
2024-08-20 03:17:00 [Parachain] err=Error { inner: Proposing
Caused by:
    0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x4f44e1 - <unknown>!rust_begin_unwind
           1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
           2: 0xc96d - <unknown>!core::option::expect_failed::hd82c6d6933db2709
           3: 0x1b22a5 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h1aa7f8bd1b00da91
           4: 0x3039ce - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::he97fed081739f443
           5: 0x3f18eb - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::h06fe1d0a212c2853
           6: 0x3f1a24 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h752f7709804d2017
           7: 0x3ac256 - <unknown>!BlockBuilder_finalize_block
    1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x4f44e1 - <unknown>!rust_begin_unwind
           1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
           2: 0xc96d - <unknown>!core::option::expect_failed::hd82c6d6933db2709
           3: 0x1b22a5 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h1aa7f8bd1b00da91
           4: 0x3039ce - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::he97fed081739f443
           5: 0x3f18eb - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::h06fe1d0a212c2853
           6: 0x3f1a24 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h752f7709804d2017
           7: 0x3ac256 - <unknown>!BlockBuilder_finalize_block }
2024-08-20 03:17:03 [Parachain] 💤 Idle (3 peers), best: #3377016 (0x3791…2dde), finalized #3377016 (0x3791…2dde), ⬇ 2.2kiB/s ⬆ 0.2kiB/s
2024-08-20 03:17:04 [Relaychain] 💤 Idle (9 peers), best: #22167599 (0x8579…f744), finalized #22167595 (0x3850…4754), ⬇ 88.8kiB/s ⬆ 208.5kiB/s
2024-08-20 03:17:06 [Relaychain] ✨ Imported #22167600 (0x197c…4ce1)
2024-08-20 03:17:06 [Parachain] 🙌 Starting consensus session on top of parent 0x3791a0253e1dbd448912bc79f38af78ffa699099c8eec346cf5232716fd12dde
2024-08-20 03:17:06 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-38f315c8db144fd3/5811785/cumulus/pallets/aura-ext/src/consensus_hook.rs:69:9:
assertion `left == right` failed: slot number mismatch
  left: Slot(143676985)
 right: Slot(287353971)
2024-08-20 03:17:06 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:06 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:06 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4f44e1 - <unknown>!rust_begin_unwind
    1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0xcd17 - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x247e96 - <unknown>!core::panicking::assert_failed::h1620870d51024939
    4: 0x27cc08 - <unknown>!frame_support::storage::transactional::with_transaction::h879f21777e24b6eb
    5: 0x43c821 - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h6fce7722202d860e
    6: 0x44646a - <unknown>!frame_support::dispatch_context::run_in_context::h04b55aceefdb7344
    7: 0x70b26 - <unknown>!<darwinia_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h9a64b8b9f9986198
    8: 0x66e57 - <unknown>!<darwinia_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h681bfff0c0c825b3
    9: 0x2ee377 - <unknown>!<fp_self_contained::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra,SelfContainedSignedInfo> as sp_runtime::traits::Applyable>::apply::h5000ae82748e94a6
   10: 0x3f14bc - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::haf6b3ddee5a7c9b9
   11: 0x3ac216 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.
2024-08-20 03:17:06 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-38f315c8db144fd3/5811785/substrate/frame/aura/src/lib.rs:416:9:
assertion `left == right` failed: Timestamp slot must match `CurrentSlot`
  left: Slot(143676985)
 right: Slot(287353971)
2024-08-20 03:17:06 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:06 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-08-20 03:17:06 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4f44e1 - <unknown>!rust_begin_unwind
    1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0xcd17 - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x247e96 - <unknown>!core::panicking::assert_failed::h1620870d51024939
    4: 0x278ea3 - <unknown>!frame_support::storage::transactional::with_transaction::h7de79e7af2ee1a2b
    5: 0x4478db - <unknown>!frame_support::dispatch_context::run_in_context::h1ed7a2444526a551
    6: 0x70b87 - <unknown>!<darwinia_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h9a64b8b9f9986198
    7: 0x66e57 - <unknown>!<darwinia_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h681bfff0c0c825b3
    8: 0x2ee377 - <unknown>!<fp_self_contained::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra,SelfContainedSignedInfo> as sp_runtime::traits::Applyable>::apply::h5000ae82748e94a6
    9: 0x3f14bc - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::haf6b3ddee5a7c9b9
   10: 0x3ac216 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.
2024-08-20 03:17:06 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-38f315c8db144fd3/5811785/cumulus/pallets/parachain-system/src/lib.rs:265:18:
set_validation_data inherent needs to be present in every block!
2024-08-20 03:17:06 [Parachain] err=Error { inner: Proposing
Caused by:
    0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x4f44e1 - <unknown>!rust_begin_unwind
           1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
           2: 0xc96d - <unknown>!core::option::expect_failed::hd82c6d6933db2709
           3: 0x1b22a5 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h1aa7f8bd1b00da91
           4: 0x3039ce - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::he97fed081739f443
           5: 0x3f18eb - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::h06fe1d0a212c2853
           6: 0x3f1a24 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h752f7709804d2017
           7: 0x3ac256 - <unknown>!BlockBuilder_finalize_block
    1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x4f44e1 - <unknown>!rust_begin_unwind
           1: 0xbcfe - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
           2: 0xc96d - <unknown>!core::option::expect_failed::hd82c6d6933db2709
           3: 0x1b22a5 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h1aa7f8bd1b00da91
           4: 0x3039ce - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::he97fed081739f443
           5: 0x3f18eb - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::h06fe1d0a212c2853
           6: 0x3f1a24 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h752f7709804d2017
           7: 0x3ac256 - <unknown>!BlockBuilder_finalize_block }
2024-08-20 03:17:08 [Parachain] 💤 Idle (3 peers), best: #3377016 (0x3791…2dde), finalized #3377016 (0x3791…2dde), ⬇ 72 B/s ⬆ 0.1kiB/s
2024-08-20 03:17:09 [Relaychain] 💤 Idle (9 peers), best: #22167600 (0x197c…4ce1), finalized #22167596 (0x59ae…8a10), ⬇ 175.0kiB/s ⬆ 223.4kiB/s
darwinia-network / darwinia

error log on collator before restarting on runtime 6550 upgrade #1570
