[Parachain Onboarding | Slot Request] ParaId: 2101

maltekliemann commented 2 months ago

Parachain Name

Zeitgeist Battery Station

ParaID

2101

Parachain Manager Account

5HGNBoNqrDayk7WdHX7eM4LApWHhW8xYBn4JnztYSwFbcbaN

Upload your Genesis State - Do not submit a compressed file.

battery_station_genesis_head.json

Upload your Validation Code (genesis runtime Wasm) - Do not submit a compressed file.

battery_station_genesis_wasm.json

maltekliemann commented 2 months ago

I think we'll continue to use the old branding. I'm assuming that means we don't need to create a PR into polkadot-js?

maltekliemann commented 2 months ago

Best regards from Team Zeitgeist 🔮

hbulgarini commented 2 months ago

Hello @maltekliemann !

I think we'll continue to use the old branding. I'm assuming that means we don't need to create a PR into polkadot-js?

Yes, you will need to open a PolkadotJS PR to migrate your settings from Rococo to Paseo.

maltekliemann commented 2 months ago

Thanks @hbulgarini! It's here: https://github.com/polkadot-js/apps/pull/10888

al3mart commented 2 months ago

maltekliemann commented 2 months ago

I'm getting the following error from our collator node:

2024-09-05 09:35:31 [Relaychain] 🔍 Discovered new external address for our node: /ip4/139.162.147.124/tcp/30336/ws/p2p/12D3KooWE8iMr17k3rKM4zKkvP2YQ3CeJWuPm4cWrh1U8PJgV6za    
2024-09-05 09:35:32 [Relaychain] Received finalized block via RPC: #2818794 (0xfadd…d9b7 -> 0xb4b4…55aa)
2024-09-05 09:35:33 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2a01:7e01::f03c:92ff:fe8a:8514/tcp/30336/ws/p2p/12D3KooWE8iMr17k3rKM4zKkvP2YQ3CeJWuPm4cWrh1U8PJgV6za    
2024-09-05 09:35:36 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 1.3kiB/s ⬆ 1.3kiB/s    
2024-09-05 09:35:36 [Relaychain] Received imported block via RPC: #2818797 (0xad52…7a05 -> 0x9203…6084)
2024-09-05 09:35:36 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x920367a6a237527f36cebd064fb79db41ca99b6c1ebaf33b96b079d350e56084 at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:35:36 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:35:41 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.1kiB/s ⬆ 0.1kiB/s    
2024-09-05 09:35:41 [Relaychain] Received finalized block via RPC: #2818795 (0xb4b4…55aa -> 0x3940…a4b5)
2024-09-05 09:35:42 [Relaychain] Received imported block via RPC: #2818798 (0x9203…6084 -> 0x296e…8bfe)
2024-09-05 09:35:42 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x296e422507e4acac4df91c1271623a51e9705d966686a2bfcc6c369e47508bfe at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:35:42 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:35:43 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-05 09:35:45 [Relaychain] Received finalized block via RPC: #2818796 (0x3940…a4b5 -> 0xad52…7a05)
2024-09-05 09:35:46 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.1kiB/s ⬆ 0.1kiB/s    
2024-09-05 09:35:48 [Relaychain] Received imported block via RPC: #2818799 (0x296e…8bfe -> 0xe1c5…165a)
2024-09-05 09:35:48 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0xe1c51ccb586a0bc42c597a2dbdfc3f16606548a6ccbbc02b44fd257da027165a at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:35:48 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:35:48 [Relaychain] Received imported block via RPC: #2818799 (0x296e…8bfe -> 0x2f4c…ddc9)
2024-09-05 09:35:48 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x2f4cf190e9b2e106ec858d0c913c38c53d49b4e88eae6439cccaa9a5a2f4ddc9 at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:35:48 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:35:51 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.2kiB/s ⬆ 0.2kiB/s    
2024-09-05 09:35:51 [🔮 Zeitgeist Parachain] ❌ Error while dialing /dns/telemetry.polkadot.io/tcp/443/x-parity-wss/%2Fsubmit%2F: Custom { kind: Other, error: Timeout }    
2024-09-05 09:35:51 [🔮 Zeitgeist Parachain] ❌ Error while dialing /dns/telemetry.zeitgeist.pm/tcp/443/x-parity-wss/%2Fsubmit%2F: Custom { kind: Other, error: Timeout }    
2024-09-05 09:35:53 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-05 09:35:53 [Relaychain] Received finalized block via RPC: #2818797 (0xad52…7a05 -> 0x9203…6084)
2024-09-05 09:35:54 [Relaychain] Received imported block via RPC: #2818800 (0xe1c5…165a -> 0xf7ce…2c95)
2024-09-05 09:35:54 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0xf7cee7044d714f067c52bfc6824b018f22a4faecaeac40734a8f604e61842c95 at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:35:54 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:35:54 [Relaychain] Received imported block via RPC: #2818800 (0xe1c5…165a -> 0xcfba…05ed)
2024-09-05 09:35:54 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0xcfba68aac816558974fec7291a9c982e16b3dc2cf107f60cab479d01d41805ed at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:35:54 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:35:56 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.2kiB/s ⬆ 0.2kiB/s    
2024-09-05 09:35:57 [Relaychain] Received finalized block via RPC: #2818798 (0x9203…6084 -> 0x296e…8bfe)
2024-09-05 09:36:00 [Relaychain] Received imported block via RPC: #2818801 (0xf7ce…2c95 -> 0xdd41…d339)
2024-09-05 09:36:00 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0xdd41974d352c70b62108421f705f2d9290cd61200166f8acfd6b313fa101d339 at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-05 09:36:00 [🔮 Zeitgeist Parachain] 🔏 No Nimbus keys available. We will not be able to author.    
2024-09-05 09:36:01 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0 ⬆ 0    
2024-09-05 09:36:03 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.

We went with the re-genesis approach. Just to walk you through what we did:

./zeitgeist export-state --chain=battery_station 5380680 > battery_station_on_paseo_spec.json
Removed key 0x45323df7cc47150b3930e2666b0aa313a2bca190d36bd834cc73a38fc213ecbd from the chain spec.
./zeitgeist export-genesis-state --chain battery_station_on_paseo_spec.json > battery_station_on_paseo_genesis_head.json
./zeitgeist export-genesis-wasm --chain battery_station_on_paseo_spec.json > battery_station_on_paseo_genesis_wasm.json
Changed local Rococo node to Paseo and synced.
Backup'ed old Battery Station DB and point all nodes to new and empty DB
Replaced --chain=battery_station with --chain=battery_station_on_paseo_spec.json in bootnode and collator calls.

I'd wager the Nimbus thing is a problem in our local setup, but "2024-09-05 09:36:03 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting." sounds like a general problem with the chain spec/state.

maltekliemann commented 2 months ago

I've also changed "relay_chain": "rococo" to "relay_chain": "paseo" in the chain spec of the parachain node. The guide doesn't mention that, but I guess that's necessary. Doesn't fix the genesis mismatch though.

al3mart commented 2 months ago

The logs also show a message reporting that the keys for nimbus are missing. I'm not super familiar with nimbus, but make sure you have imported the right keys into the node.

Regarding 2024-09-05 09:36:03 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting. It seems the node has discovered a node running a paseo network with a different spec. That's why it's banning it. Nothing alarming.

maltekliemann commented 2 months ago

We most likely need to register the genesis in the relay again.

Why though? What went wrong?

Could you export the genesis state and wasm again and share that here ? We will update the relay state.

The CurrentCodeHash on Paseo is the same as the one on Rococo, which makes sense, but the result of b2sum -l 256 battery_station_genesis_wasm.json is different. I'm honestly not sure what to expect here, but this is surprising to me.

The logs also show a message reporting that the keys for nimbus are missing.

Yeah, I'm working on fixing that.

It seems the node has discovered a node running a paseo network with a different spec. That's why it's banning it. Nothing alarming.

Now I'm extra confused. So you're saying the genesis mismatch is not caused by a problem with the data uploaded to the relaychain?

I'm running a local Paseo node and using that as relay chain RPC for the parachain node. I had to manually export the Paseo spec and then specify that spec when running the parachain node. Not sure why, but I guess our parachain code is too old to know about Paseo. Is that what's causing that?

What I'm hearing is that the Nimbus issue seems to be the only thing in the way of us actually producing blocks...?

maltekliemann commented 2 months ago

Okay, after inserting the Nimbus keys, I'm now getting this:

2024-09-06 09:18:48 [🔮 Zeitgeist Parachain] 💤 Idle (0 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 68 B/s ⬆ 0.1kiB/s    
2024-09-06 09:18:53 [Relaychain] Received finalized block via RPC: #2833021 (0x3963…35c0 -> 0xe8de…d14f)
2024-09-06 09:18:53 [🔮 Zeitgeist Parachain] 💤 Idle (0 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 12 B/s ⬆ 12 B/s    
2024-09-06 09:18:54 [Relaychain] Received imported block via RPC: #2833024 (0x3ec4…9c6d -> 0x4925…4f3b)
2024-09-06 09:18:54 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x49253fb63f3ec01325ee2b79b5b6e87cc73903ceabee96cffc494e138c674f3b at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-06 09:18:54 [🔮 Zeitgeist Parachain] 🔮 Skipping candidate production because we are not eligible for slot 2833024    
2024-09-06 09:18:57 [Relaychain] Received finalized block via RPC: #2833022 (0xe8de…d14f -> 0x45fe…effa)
2024-09-06 09:18:58 [🔮 Zeitgeist Parachain] 💤 Idle (0 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.3kiB/s ⬆ 0.4kiB/s    
2024-09-06 09:19:00 [Relaychain] Received imported block via RPC: #2833025 (0x4925…4f3b -> 0x4dac…420f)
2024-09-06 09:19:00 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x4dac0fa74a5a68a8cd870f22a53ec8b536cae8551fa5de2c4cc5aa64c57f420f at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-06 09:19:00 [🔮 Zeitgeist Parachain] 🔮 Skipping candidate production because we are not eligible for slot 2833025    
2024-09-06 09:19:03 [🔮 Zeitgeist Parachain] 💤 Idle (0 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 12 B/s ⬆ 43 B/s    
2024-09-06 09:19:05 [Relaychain] Received finalized block via RPC: #2833023 (0x45fe…effa -> 0x3ec4…9c6d)
2024-09-06 09:19:06 [Relaychain] Received imported block via RPC: #2833026 (0x4dac…420f -> 0x1a77…f109)
2024-09-06 09:19:06 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x1a77d2f6172dae6b3603512c3d8505b284d8dfe86005e0c5f6467c360f3cf109 at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-06 09:19:06 [🔮 Zeitgeist Parachain] 🔮 Skipping candidate production because we are not eligible for slot 2833026    
2024-09-06 09:19:08 [🔮 Zeitgeist Parachain] Report 12D3KooWFJot9umYMEMrTDXvnnpcC7hMsaJ6GxeGQLqen3ViDfxy: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-06 09:19:08 [🔮 Zeitgeist Parachain] 💤 Idle (0 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.2kiB/s ⬆ 0.2kiB/s    
2024-09-06 09:19:09 [Relaychain] Received finalized block via RPC: #2833024 (0x3ec4…9c6d -> 0x4925…4f3b)
2024-09-06 09:19:12 [Relaychain] Received imported block via RPC: #2833027 (0x1a77…f109 -> 0x7f83…fccc)
2024-09-06 09:19:12 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x7f838469a16adcf8f09cc3b462361e1e95da751c701b67d403b38b7aea33fccc at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-06 09:19:12 [🔮 Zeitgeist Parachain] 🔮 Skipping candidate production because we are not eligible for slot 2833027

al3mart commented 2 months ago

Hey @maltekliemann I have pushed the new state and code for your parachain. It seems the parachain is still not producing any blocks. The logs you are getting don't really provide us with much context on what is that it might be happening.

Activating some more logs related to nimbus should be helpful. Maybe some storage needs to be cleaned if the authorities have changed. Please, also notice that I'm just guessing what might be happening. As I said, I'm not super familiar with the details of nimbus.

maltekliemann commented 2 months ago

Hey @maltekliemann I have pushed the new state and code for your parachain. It seems the parachain is still not producing any blocks. The logs you are getting don't really provide us with much context on what is that it might be happening.

I'm still puzzled about the current code hash you're using:

Is the current code hash expected to change when going through the re-genesis process?
The current code hash on Paseo is the same as the one on Rococo, which makes sense, but the result of b2sum -l 256 battery_station_genesis_wasm.json is different. Is this expected?

Activating some more logs related to nimbus should be helpful. Maybe some storage needs to be cleaned if the authorities have changed. Please, also notice that I'm just guessing what might be happening. As I said, I'm not super familiar with the details of nimbus.

Yeah, nimbus is a bit of a blackbox, I'm afraid. The authorities haven't changed though. I'm using the same Nimbus keys as before, so no storage changes should be necessary. I'm hoping that it's just a key problem on our side, though, but I think we need to get some clarity into what the current code hash ought to be.

maltekliemann commented 2 months ago

Hey @maltekliemann I have pushed the new state and code for your parachain.

Or, to ask a slightly different question, what's the difference between the state and code before and after you pushed it a second time?

maltekliemann commented 2 months ago

I managed to just brute-force the keys of the old node over to the new node, now I'm getting this:

2024-09-11 10:21:48 [Relaychain] Received imported block via RPC: #2905560 (0x6276…04a1 -> 0x9bc7…abb4)
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 💤 Idle (1 peers), best: #0 (0x6648…9213), finalized #0 (0x6648…9213), ⬇ 0.7kiB/s ⬆ 0.7kiB/s    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 🙌 Starting consensus session on top of parent 0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/c8d2251/cumulus/pallets/parachain-system/src/lib.rs:1100:9:
assertion `left == right` failed
  left: 0x034acd338fd6767f14242f0779a3b239b47f7e075c51585370f163c8f3ced652
 right: 0x0000000000000000000000000000000000000000000000000000000000000000    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x506a65 - <unknown>!rust_begin_unwind
    1: 0x2d97 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x21023 - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x165614 - <unknown>!core::panicking::assert_failed::h9f5d93bb7b94c3e2
    4: 0xdc34b - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h190f619a51c33124
    5: 0x58158 - <unknown>!<battery_station_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::hf791263c80dbdfd7
    6: 0x50691 - <unknown>!<battery_station_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h4dcbf43cb2c1d19d
    7: 0x42eb8a - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::ha86ef34f094ed726
    8: 0x466267 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] panicked at /home/builder/cargo/git/checkouts/moonkit-108dd061c923821f/34c21d6/pallets/author-inherent/src/lib.rs:130:13:
Block invalid; Supplied slot number is not high enough    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x506a65 - <unknown>!rust_begin_unwind
    1: 0x2d97 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0xc974d - <unknown>!<pallet_author_inherent::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h819870892d99a8a3
    3: 0x563bc - <unknown>!<battery_station_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::hf791263c80dbdfd7
    4: 0x50691 - <unknown>!<battery_station_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h4dcbf43cb2c1d19d
    5: 0x42eb8a - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::ha86ef34f094ed726
    6: 0x466267 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/c8d2251/cumulus/pallets/parachain-system/src/lib.rs:248:13:
set_validation_data inherent needs to be present in every block!    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] Proposing failed. error=RuntimeApiError(Application(Execution(AbortedDueToTrap(MessageWithBacktrace { message: "wasm trap: wasm `unreachable` instruction executed", backtrace: Some(Backtrace { backtrace_string: "error while executing at wasm backtrace:\n    0: 0x506a65 - <unknown>!rust_begin_unwind\n    1: 0x2d97 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577\n    2: 0x43a1e1 - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hc170ebb09ef42546\n    3: 0x42f311 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::h5d39ce2d610a3c06\n    4: 0x468509 - <unknown>!BlockBuilder_finalize_block" }) }))))
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x9bc74f6b5468028a9cc6a75791b9149b1e18a048bc198c05def7e43d5e75abb4 at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] 🙌 Starting consensus session on top of parent 0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213    
2024-09-11 10:21:49 [🔮 Zeitgeist Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/c8d2251/cumulus/pallets/parachain-system/src/lib.rs:1100:9:
assertion `left == right` failed

maltekliemann commented 2 months ago

Seems like the expected DMQ head is zero.

al3mart commented 2 months ago

I have modified the state on the relay chain. Please, let me know how the logs are looking now.

maltekliemann commented 2 months ago

Now getting this

2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] 🎁 Prepared block for proposing at 1 (0 ms) [hash: 0x2fc4ffb2a554059ceb5929f7751dbb6784f4972f313c2ad8d85b5f6df5b8c52b; parent_hash: 0x6648…9213; extrinsics (2): [0xfbed…9c7a, 0x82c8…acad]    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] 🔖 Sealed block for proposal at 1. Hash now 0xa30120fe38fdec1bc84e04808dbd90a7cabcb361b14ea054fb7a5db18a41d083, previously 0x2fc4ffb2a554059ceb5929f7751dbb6784f4972f313c2ad8d85b5f6df5b8c52b.    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] ✨ Imported #1 (0xa301…d083)    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] PoV size { header: 0.2431640625kb, extrinsics: 5.4716796875kb, storage_proof: 15.392578125kb }
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] Compressed PoV size: 17.7861328125kb
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] Produced proof-of-validity candidate. block_hash=0xa30120fe38fdec1bc84e04808dbd90a7cabcb361b14ea054fb7a5db18a41d083
2024-09-12 10:09:12 [Relaychain] Received imported block via RPC: #2919827 (0xb658…b4ce -> 0x7722…3dcc)
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x77226673b1707f1d39f81017ecc29b9455d7eda0498920df40d779bc07583dcc at=0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] 🙌 Starting consensus session on top of parent 0x6648ad634c073f1a24f3323943952488e4f8d0e2226288b3a3d52f70564f9213    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] panicked at /home/builder/cargo/git/checkouts/moonkit-108dd061c923821f/34c21d6/pallets/author-inherent/src/lib.rs:130:13:
Block invalid; Supplied slot number is not high enough    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-09-12 10:09:12 [🔮 Zeitgeist Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x506a65 - <unknown>!rust_begin_unwind
    1: 0x2d97 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0xc974d - <unknown>!<pallet_author_inherent::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h819870892d99a8a3
    3: 0x563bc - <unknown>!<battery_station_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::hf791263c80dbdfd7
    4: 0x50691 - <unknown>!<battery_station_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h4dcbf43cb2c1d19d
    5: 0x42eb8a - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::ha86ef34f094ed726
    6: 0x466267 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.

The problem is that the HighestSlotSeen of author-inherent is too large. So I've killed that storage item in the chain spec and re-exported the genesis head:

battery_station_on_paseo_sans_highest_slot_seen_genesis_state.json

Could you please set this as the current head? I think that might cut it.

maltekliemann commented 2 months ago

(WASM stays the same, of course.)

al3mart commented 2 months ago

@maltekliemann the new state is now in, let me know if any DMQ heads logs are showing up and need addressing on our side.

maltekliemann commented 2 months ago

Looks like we're producing now! ⛏️

2024-09-16 11:40:12 [🔮 Zeitgeist Parachain] Starting collation. relay_parent=0x7e95b227b3d89dba600ccbf91833dbc34622ca803559f580c47c47c0e17b154f at=0xfb2f6c0837c11d62c3554fc042b644563e3be9362efeddf63e95042607a3904f
2024-09-16 11:40:13 [🔮 Zeitgeist Parachain] 🙌 Starting consensus session on top of parent 0xfb2f6c0837c11d62c3554fc042b644563e3be9362efeddf63e95042607a3904f    
2024-09-16 11:40:13 [🔮 Zeitgeist Parachain] 🎁 Prepared block for proposing at 1 (1 ms) [hash: 0x43d96920dd48773f43afc84d990452721c791da522c64019ad004cc72b5430df; parent_hash: 0xfb2f…904f; extrinsics (3): [0x190a…1357, 0x9176…7c2c, 0x0eb6…531d]    
2024-09-16 11:40:13 [🔮 Zeitgeist Parachain] 🔖 Sealed block for proposal at 1. Hash now 0x2b44e23da249fa65ed41e8ddb09da9aaf7612ca5377258e122b59c7b043442b9, previously 0x43d96920dd48773f43afc84d990452721c791da522c64019ad004cc72b5430df.    
2024-09-16 11:40:13 [🔮 Zeitgeist Parachain] ✨ Imported #1 (0x2b44…42b9)    
2024-09-16 11:40:13 [🔮 Zeitgeist Parachain] PoV size { header: 0.2431640625kb, extrinsics: 5.48828125kb, storage_proof: 15.443359375kb }
2024-09-16 11:40:13 [🔮 Zeitgeist Parachain] Compressed PoV size: 17.8359375kb

Thanks for all the support, very much appreciated!

maltekliemann commented 2 months ago

Actually, can you do a complete reset? We had a bit of a hiccup with our database. Can you just reset us to block zero again?

al3mart commented 2 months ago

Same head than here ?

https://github.com/paseo-network/support/issues/93#issuecomment-2345917808

I've also removed the manager lock on your para so you can set the necessary head for it with the manager account as needed.

maltekliemann commented 2 months ago

Yes, please 🙏

maltekliemann commented 2 months ago

2024-09-19 16:28:39 [Relaychain] Received finalized block via RPC: #3024238 (0x3c18…d5d0 -> 0xba7c…7014)
2024-09-19 16:28:40 [🔮 Zeitgeist Parachain] 💤 Idle (2 peers), best: #0 (0xfb2f…904f), finalized #0 (0xfb2f…904f), ⬇ 0.5kiB/s ⬆ 0.5kiB/s    
2024-09-19 16:28:42 [Relaychain] Received imported block via RPC: #3024241 (0x64ac…f00f -> 0x45f1…68e7)
2024-09-19 16:28:42 [🔮 Zeitgeist Parachain] Report 12D3KooWFJot9umYMEMrTDXvnnpcC7hMsaJ6GxeGQLqen3ViDfxy: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-19 16:28:45 [🔮 Zeitgeist Parachain] 💤 Idle (2 peers), best: #0 (0xfb2f…904f), finalized #0 (0xfb2f…904f), ⬇ 0.2kiB/s ⬆ 0.2kiB/s    
2024-09-19 16:28:47 [Relaychain] Received finalized block via RPC: #3024239 (0xba7c…7014 -> 0xce0a…d403)
2024-09-19 16:28:47 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-19 16:28:48 [Relaychain] Received imported block via RPC: #3024242 (0x45f1…68e7 -> 0xec05…c0fe)
2024-09-19 16:28:50 [🔮 Zeitgeist Parachain] 💤 Idle (2 peers), best: #0 (0xfb2f…904f), finalized #0 (0xfb2f…904f), ⬇ 0.1kiB/s ⬆ 0.2kiB/s    
2024-09-19 16:28:51 [Relaychain] Received finalized block via RPC: #3024240 (0xce0a…d403 -> 0x64ac…f00f)
2024-09-19 16:28:52 [🔮 Zeitgeist Parachain] Report 12D3KooWFJot9umYMEMrTDXvnnpcC7hMsaJ6GxeGQLqen3ViDfxy: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-19 16:28:54 [Relaychain] Received imported block via RPC: #3024243 (0xec05…c0fe -> 0xea8c…359a)
2024-09-19 16:28:55 [🔮 Zeitgeist Parachain] 💤 Idle (2 peers), best: #0 (0xfb2f…904f), finalized #0 (0xfb2f…904f), ⬇ 71 B/s ⬆ 56 B/s    
2024-09-19 16:28:57 [🔮 Zeitgeist Parachain] Report 12D3KooWF6t8jjh4ws8MeKc34XBXzJxFcbVJ94XaBz1gxd7KS3mD: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-19 16:28:59 [Relaychain] Received finalized block via RPC: #3024241 (0x64ac…f00f -> 0x45f1…68e7)
2024-09-19 16:29:00 [Relaychain] Received imported block via RPC: #3024244 (0xea8c…359a -> 0x63aa…7161)
2024-09-19 16:29:00 [🔮 Zeitgeist Parachain] 💤 Idle (2 peers), best: #0 (0xfb2f…904f), finalized #0 (0xfb2f…904f), ⬇ 0.3kiB/s ⬆ 0.3kiB/s    
2024-09-19 16:29:02 [🔮 Zeitgeist Parachain] Report 12D3KooWFJot9umYMEMrTDXvnnpcC7hMsaJ6GxeGQLqen3ViDfxy: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.    
2024-09-19 16:29:03 [Relaychain] Received finalized block via RPC: #3024242 (0x45f1…68e7 -> 0xec05…c0fe)

No error messages, except for the genesis mismatch from earlier, but it's not producing any blocks. Are you sure you've set everything back to the original state? Seems like we're doing everything we did before, but now it's not working.

al3mart commented 2 months ago

Everything is set an block production is back now :+1: I'll go ahead and close this issue. Thanks for your patience during the onboarding.

paseo-network / support