Collator Node panicked, then restarted

[dcolley]

Describe the bug A clear and concise description of what the bug is.

│2024-09-17T08:09:06.248145+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:06 [Relaychain] ✨ Imported #22572512 (0x9700…b89a)                                                                                                  │
│2024-09-17T08:09:06.348101+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:06 [Relaychain] 💤 Idle (11 peers), best: #22572512 (0x9700…b89a), finalized #22572500 (0x1d42…3a62), ⬇ 61.0kiB/s ⬆ 217.2kiB/s                       │
│2024-09-17T08:09:06.398198+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:06 💤 Idle (23 peers), best: #6982477 (0x4aa0…2eb6), finalized #6982475 (0x90ad…9477), ⬇ 1.3kiB/s ⬆ 1.7kiB/s                                         │
│2024-09-17T08:09:11.355901+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:11 [Relaychain] 💤 Idle (11 peers), best: #22572512 (0x9700…b89a), finalized #22572500 (0x1d42…3a62), ⬇ 91.7kiB/s ⬆ 140.6kiB/s                       │
│2024-09-17T08:09:11.405708+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:11 💤 Idle (37 peers), best: #6982477 (0x4aa0…2eb6), finalized #6982475 (0x90ad…9477), ⬇ 6.0kiB/s ⬆ 6.7kiB/s                                         │
│2024-09-17T08:09:12.357359+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:12 [Relaychain] ✨ Imported #22572513 (0x086f…974f)                                                                                                  │
│2024-09-17T08:09:15.565850+00:00 aca-collator-1 qemu-ga: info: guest-ping called                                                                                                                                                     │
│2024-09-17T08:09:16.363633+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:16 [Relaychain] 💤 Idle (11 peers), best: #22572513 (0x086f…974f), finalized #22572500 (0x1d42…3a62), ⬇ 127.0kiB/s ⬆ 154.6kiB/s                      │
│2024-09-17T08:09:16.369597+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:16 💤 Idle (39 peers), best: #6982477 (0x4aa0…2eb6), finalized #6982475 (0x90ad…9477), ⬇ 1.1kiB/s ⬆ 1.9kiB/s                                         │
│2024-09-17T08:09:16.369645+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:16 ❌ Error while dialing /dns/telemetry.w3f.community/tcp/443/x-parity-wss/%2Fsubmit: Custom { kind: Other, error: Other(Left(Handshake("server reje│
ted handshake; status code = 404"))) }                                                                                                                                                                                                │
│2024-09-17T08:09:18.166278+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:18 ✨ Imported #6982478 (0x4890…2d87)                                                                                                                │
│2024-09-17T08:09:21.371151+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:21 [Relaychain] 💤 Idle (11 peers), best: #22572513 (0x086f…974f), finalized #22572500 (0x1d42…3a62), ⬇ 77.0kiB/s ⬆ 130.9kiB/s                       │
│2024-09-17T08:09:21.371295+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:21 💤 Idle (36 peers), best: #6982478 (0x4890…2d87), finalized #6982475 (0x90ad…9477), ⬇ 17.2kiB/s ⬆ 53.6kiB/s                                       │
│2024-09-17T08:09:24.440145+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 [Relaychain] ✨ Imported #22572514 (0x6039…2162)                                                                                                  │
│2024-09-17T08:09:24.440335+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 🙌 Starting consensus session on top of parent 0x489081d4e1ac796b960e80692297ac026bcbdbe168b9a29d73a5d4bb9d5f2d87                                 │
│2024-09-17T08:09:24.440402+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.10.0/src/lib.rs:1285:17: 
2024-09-17T08:09:24.440452+00:00 aca-collator-1 acala[179904]: assertion `left == right` failed: expected parent to be included                                                                                                      │
│2024-09-17T08:09:24.440504+00:00 aca-collator-1 acala[179904]:   left: 0x4aa05cbaf64d752f2fc1ed900954b95d0ff246a6eb39d5eb6f0331de1cc22eb6                                                                                            │
│2024-09-17T08:09:24.440599+00:00 aca-collator-1 acala[179904]:  right: 0x489081d4e1ac796b960e80692297ac026bcbdbe168b9a29d73a5d4bb9d5f2d87                                                                                            │
│2024-09-17T08:09:24.440667+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 1 storage transactions are left open by the runtime. Those will be rolled back.                                                                   │
│2024-09-17T08:09:24.440718+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 1 storage transactions are left open by the runtime. Those will be rolled back.                                                                   │
│2024-09-17T08:09:24.440757+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `u│
nreachable` instruction executed                                                                                                                                                                                                      │
│2024-09-17T08:09:24.440802+00:00 aca-collator-1 acala[179904]: WASM backtrace:                                                                                                                                                       │
│2024-09-17T08:09:24.440845+00:00 aca-collator-1 acala[179904]: error while executing at wasm backtrace:                                                                                                                              │
│2024-09-17T08:09:24.440911+00:00 aca-collator-1 acala[179904]:     0: 0x569e62 - <unknown>!rust_begin_unwind                                                                                                                         │
│2024-09-17T08:09:24.440954+00:00 aca-collator-1 acala[179904]:     1: 0x6902 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577                                                                                               │
│2024-09-17T08:09:24.440989+00:00 aca-collator-1 acala[179904]:     2: 0x14755 - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca                                                                                    │
│2024-09-17T08:09:24.441019+00:00 aca-collator-1 acala[179904]:     3: 0x1f8b5b - <unknown>!core::panicking::assert_failed::h39c657b1c58e7b76                                                                                         │
│2024-09-17T08:09:24.441058+00:00 aca-collator-1 acala[179904]:     4: 0x1effb6 - <unknown>!cumulus_pallet_parachain_system::pallet::Pallet<T>::set_validation_data::{{closure}}::hf99d88e6cbfc3ca8                                   │
│2024-09-17T08:09:24.441103+00:00 aca-collator-1 acala[179904]:     5: 0x1ec4a0 - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{c│
losure}}::hcea9fc3581419e98                                                                                                                                                                                                           │
│2024-09-17T08:09:24.441141+00:00 aca-collator-1 acala[179904]:     6: 0x142fdf - <unknown>!frame_support::dispatch_context::run_in_context::hf41be5553f159799                                                                        │
│2024-09-17T08:09:24.441178+00:00 aca-collator-1 acala[179904]:     7: 0x10ee70 - <unknown>!<acala_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::hc1189227691748ad        │
│2024-09-17T08:09:24.441228+00:00 aca-collator-1 acala[179904]:     8: 0x453a0f - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h145d795bcbe68│
e91                                                                                                                                                                                                                                   │
│2024-09-17T08:09:24.441268+00:00 aca-collator-1 acala[179904]:     9: 0x4b7d30 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.                                                                                                   │
│2024-09-17T08:09:24.441294+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.10.0/src/lib.rs:266:18:           │
│2024-09-17T08:09:24.441322+00:00 aca-collator-1 acala[179904]: set_validation_data inherent needs to be present in every block!                                                                                                      │
│2024-09-17T08:09:24.441346+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:24 err=Error { inner: Proposing                                                                                                                      │
│2024-09-17T08:09:24.441377+00:00 aca-collator-1 acala[179904]: Caused by:                                                                                                                                                            │
│2024-09-17T08:09:24.441407+00:00 aca-collator-1 acala[179904]:     0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed                              │
│2024-09-17T08:09:24.441431+00:00 aca-collator-1 acala[179904]:        WASM backtrace:                                                                                                                                                │
│2024-09-17T08:09:24.441462+00:00 aca-collator-1 acala[179904]:        error while executing at wasm backtrace:                                                                                                                       │
│2024-09-17T08:09:24.441488+00:00 aca-collator-1 acala[179904]:            0: 0x569e62 - <unknown>!rust_begin_unwind                                                                                                                  │
│2024-09-17T08:09:24.441510+00:00 aca-collator-1 acala[179904]:            1: 0x6902 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577                                                                                        │
│2024-09-17T08:09:24.441534+00:00 aca-collator-1 acala[179904]:            2: 0x35a7 - <unknown>!core::option::expect_failed::hd82c6d6933db2709                                                                                       │
│2024-09-17T08:09:24.441571+00:00 aca-collator-1 acala[179904]:            3: 0x471806 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::Hooks<<<<T as frame_system::pallet::Config>::B│
lock as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::hb55177ef52382a28                                                                                                         │
│2024-09-17T08:09:24.441621+00:00 aca-collator-1 acala[179904]:            4: 0x456297 - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,Tup│
leElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElem│
ent24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39│
,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,Tupl│
eElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59,TupleElement60,TupleElement61,TupleElement62,TupleElement63,TupleElement64,TupleElement65,TupleElement66,TupleElement67,TupleElement68,TupleElement69) as frame│
_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::ha35ab407a75b482c                                                                                                                                                     │
│2024-09-17T08:09:24.441681+00:00 aca-collator-1 acala[179904]:            5: 0x4b8274 - <unknown>!BlockBuilder_finalize_block                                                                                                        │
│2024-09-17T08:09:24.441721+00:00 aca-collator-1 acala[179904]:     1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed                                                            │
│2024-09-17T08:09:24.441758+00:00 aca-collator-1 acala[179904]:        WASM backtrace:                                                                                                                                                │
│2024-09-17T08:09:24.441793+00:00 aca-collator-1 acala[179904]:        error while executing at wasm backtrace:                                                                                                                       │
│2024-09-17T08:09:24.441828+00:00 aca-collator-1 acala[179904]:            0: 0x569e62 - <unknown>!rust_begin_unwind                                                                                                                  │
│2024-09-17T08:09:24.441872+00:00 aca-collator-1 acala[179904]:            1: 0x6902 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577    
│2024-09-17T08:09:24.441910+00:00 aca-collator-1 acala[179904]:            2: 0x35a7 - <unknown>!core::option::expect_failed::hd82c6d6933db2709                                                                                       │
│2024-09-17T08:09:24.441947+00:00 aca-collator-1 acala[179904]:            3: 0x471806 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::Hooks<<<<T as frame_system::pallet::Config>::B│
lock as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::hb55177ef52382a28                                                                                                         │
│2024-09-17T08:09:24.441985+00:00 aca-collator-1 acala[179904]:            4: 0x456297 - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,Tup│
leElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElem│
ent24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39│
,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,Tupl│
eElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59,TupleElement60,TupleElement61,TupleElement62,TupleElement63,TupleElement64,TupleElement65,TupleElement66,TupleElement67,TupleElement68,TupleElement69) as frame│
_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::ha35ab407a75b482c                                                                                                                                                     │
│2024-09-17T08:09:24.442037+00:00 aca-collator-1 acala[179904]:            5: 0x4b8274 - <unknown>!BlockBuilder_finalize_block }                                                                                                      │
│2024-09-17T08:09:26.328627+00:00 aca-collator-1 acala[179904]: 2024-09-17 08:09:26 [Relaychain] 💤 Idle (11 peers), best: #22572514 (0x6039…2162), finalized #22572500 (0x1d42…3a62), ⬇ 92.5kiB/s ⬆ 230.1kiB/s  

Expected Behavior The node should not panic

Current Behavior

Steps to Reproduce ?

Additional context

• Node version: acala 2.25.0-eb961734875
• OS: Linux aca-collator-1 6.2.0-39-generic #40-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:18:00 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Command line options:

  ACALA_CLI_ARGS=\
  --chain /<..snip..>/acala-dist.json \
  --base-path /data \
  --name METASPAN \
  --collator \
  --telemetry-url 'wss://telemetry.polkadot.io/submit 1' \
  --prometheus-port 9615 \
  --prometheus-external \
  --public-addr /ip4/195.144.22.130/tcp/18022/p2p/12D3KooWNutysTNUrC6cWs3duD1zKjax3zW35jcbUH5uSV4B6XtR \
  --listen-addr /ip4/0.0.0.0/tcp/30333 \
  --state-pruning archive \
  --rpc-cors all \
  --rpc-methods safe \
  -- --chain /<..snip..>/polkadot.json \
  --base-path /data \
  --sync warp \
  --name METASPAN-ACA \
  --telemetry-url 'wss://telemetry.polkadot.io/submit 1' \
  --prometheus-port 9615 \
  --prometheus-external \
  --listen-addr /ip4/0.0.0.0/tcp/30334 \
  --state-pruning 1000 \
  --rpc-cors all \
  --rpc-methods safe

• Relevant transaction hash:
• Relevant account address:
• Logs. Please format it as code blocks

[dcolley]

Small CPU increase on the hardware:

[xlc]

is this a one time error or does it repeats after restart?

[dcolley]

Just once this morning, nothing else in the logs

[bkchr]

I think this was related to the issue on Polkadot this morning. The issue can probably be fixed @xlc. If this happens again, feel free to ping me.

[xlc]

Closing as it is a Polkadot issue and not expected to happen again.