[Parachain Onboarding | Slot Request] ParaId: 2124

gianfra-t commented 2 months ago

Parachain Name

Amplitude testnet (Foucoco)

ParaID

2124

Parachain Manager Account

5FNJjjUnb6WpyXCGB527689Ssvpiu6ZxdcypuoFrJ49XLrWK

Upload your Genesis State - Do not submit a compressed file.

parachain-genesis-head-for-paseo.txt

Upload your Validation Code (genesis runtime Wasm) - Do not submit a compressed file.

parachain-genesis-wasm-for-paseo.txt

educlerici-zondax commented 2 months ago

hi @gianfra-t is the Parachain Manager Account correct?

gianfra-t commented 2 months ago

Hi @educlerici-zondax, yes it is a newly created account, since we don't have access anymore to the same Manager Account used in Rococo. Is this okay? Also, should it get funded first?

al3mart commented 2 months ago

Hi @gianfra-t ! Just checking if your collators are running :) I've noticed that no blocks are being produced after the onboarding.

gianfra-t commented 2 months ago

They are not yet! I will inform the team so we set them up ASAP to test the integration. Thanks!

gianfra-t commented 2 months ago

Hi @al3mart, going back to the topic, we are having some issues running our new collators.

So far what we've attempted is starting running our collators with the new paseo spec we generated from the Rococo chain, and of course Paseo raw spec. We removed the old database, and also the bootnodes from the spec.

When the collator starts, we see in the logs the expected genesis state:

Parachain genesis state: 0x000000000000000000000000000000000000000000000000000000000000000000c8429ca368cded78085d5abea21ca6ef308df8b013e877b7e8ad4f9412ca680703170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400

yet after the relay finishes syncing we see the following log:

2024-09-09 11:47:32 [Parachain] 💤 Idle (0 peers), best: #0 (0x219e…ffe2), finalized #0 (0x219e…ffe2), ⬇ 0 ⬆ 0    
2024-09-09 11:47:36 [Relaychain] ✨ Imported #2879465 (0x7e9f…8904)    
2024-09-09 11:47:36 [Parachain] Could not find the header of the genesis block in the database! block_hash=0x2b5696b3842c4f144175180613feabee6940c5471d4871e94791023c94c66338

Any idea what could be going on?

For completeness, here is the exact spec file in question: parachain-spec-for-paseo.json

And just in case it's useful the complete snippet of the logs when starting the collator (error comes later after sync finishes):

2024-09-09 16:03:58 Pendulum Collator
2024-09-09 16:03:58 ✌️  version 0.1.0-27950be61f0
2024-09-09 16:03:58 ❤️  by Pendulum, 2022-2024
2024-09-09 16:03:58 📋 Chain specification: Foucoco
2024-09-09 16:03:58 🏷  Node name: pencol-roc-04
2024-09-09 16:03:58 👤 Role: AUTHORITY
2024-09-09 16:03:58 💾 Database: RocksDb at /efs_x/pencol-roc-04/chains/foucoco/db/full
2024-09-09 16:03:58 ⛓  Native runtime: foucoco-10 (foucoco-0.tx8.au1)
2024-09-09 16:04:01 Parachain id: Id(2124)
2024-09-09 16:04:01 Parachain Account: 5Ec4AhP2VP3kQcM928TnKuw9miE7yCDvRxd3mqVNA28pE3m4
2024-09-09 16:04:01 Parachain genesis state: 0x000000000000000000000000000000000000000000000000000000000000000000c8429ca368cded78085d5abea21ca6ef308df8b013e877
b7e8ad4f9412ca680703170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400
2024-09-09 16:04:01 Is collating: yes
2024-09-09 16:04:04 [Parachain] 🔨 Initializing Genesis block/state (state: 0xec8e…f88c, header-hash: 0x219e…ffe2)
2024-09-09 16:04:10 [Relaychain] 🔨 Initializing Genesis block/state (state: 0x2b2a…6ab6, header-hash: 0x77af…764f)
2024-09-09 16:04:10 [Relaychain] 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2024-09-09 16:04:12 [Relaychain] 👶 Creating empty BABE epoch changes on what appears to be first startup.
2024-09-09 16:04:12 [Relaychain] 🏷  Local node identity is: 12D3KooWFVR3LKVNwNE12ocyquY5eZNsWe3jSCTLxpvFxZgekaWo
2024-09-09 16:04:12 [Relaychain] 💻 Operating system: linux
2024-09-09 16:04:12 [Relaychain] 💻 CPU architecture: x86_64
2024-09-09 16:04:12 [Relaychain] 💻 Target environment: gnu
2024-09-09 16:04:12 [Relaychain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
2024-09-09 16:04:12 [Relaychain] 💻 CPU cores: 8
2024-09-09 16:04:12 [Relaychain] 💻 Memory: 30076MB
2024-09-09 16:04:12 [Relaychain] 💻 Kernel: 5.15.0-119-generic
2024-09-09 16:04:12 [Relaychain] 💻 Linux distribution: Ubuntu 22.04.3 LTS
2024-09-09 16:04:12 [Relaychain] 💻 Virtual machine: yes
2024-09-09 16:04:12 [Relaychain] 📦 Highest known block at #0
2024-09-09 16:04:12 [Relaychain] 〽️ Prometheus exporter started at 0.0.0.0:9616
2024-09-09 16:04:12 [Relaychain] Running JSON-RPC HTTP server: addr=127.0.0.1:9934, allowed origins=["http://localhost/:*", "http://127.0.0.1/:*", "https://local/
host:*", "https://127.0.0.1/:*", "https://polkadot.js.org/"]
2024-09-09 16:04:12 [Relaychain] Running JSON-RPC WS server: addr=127.0.0.1:9945, allowed origins=["http://localhost/:*", "http://127.0.0.1/:*", "https://localho/
st:*", "https://127.0.0.1/:*", "https://polkadot.js.org/"]
2024-09-09 16:04:12 [Relaychain] 🏁 CPU score: 783.43 MiBs
2024-09-09 16:04:12 [Relaychain] 🏁 Memory score: 6.01 GiBs
2024-09-09 16:04:12 [Relaychain] 🏁 Disk score (seq. writes): 924.06 MiBs
2024-09-09 16:04:12 [Relaychain] 🏁 Disk score (rand. writes): 81.53 MiBs
2024-09-09 16:04:12 [Relaychain] Starting with an empty approval vote DB.
2024-09-09 16:04:12 [Parachain] 🏷  Local node identity is: 12D3KooWJAfW5JEbC4SHCVh8Wdi76optU6gz6iGzqLy9UQmChmzB
2024-09-09 16:04:12 [Parachain] 💻 Operating system: linux
2024-09-09 16:04:12 [Parachain] 💻 CPU architecture: x86_64
2024-09-09 16:04:12 [Parachain] 💻 Target environment: gnu
2024-09-09 16:04:12 [Parachain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
2024-09-09 16:04:12 [Parachain] 💻 CPU cores: 8
2024-09-09 16:04:12 [Parachain] 💻 Memory: 30076MB
2024-09-09 16:04:12 [Parachain] 💻 Kernel: 5.15.0-119-generic
2024-09-09 16:04:12 [Parachain] 💻 Linux distribution: Ubuntu 22.04.3 LTS
2024-09-09 16:04:12 [Parachain] 💻 Virtual machine: yes
2024-09-09 16:04:12 [Parachain] 📦 Highest known block at #0
2024-09-09 16:04:12 [Parachain] 〽️ Prometheus exporter started at 0.0.0.0:9615
2024-09-09 16:04:12 [Parachain] Running JSON-RPC HTTP server: addr=127.0.0.1:9955, allowed origins=["*"]
2024-09-09 16:04:12 [Parachain] Running JSON-RPC WS server: addr=127.0.0.1:8844, allowed origins=["*"]
2024-09-09 16:04:12 [Parachain] 🏁 CPU score: 783.43 MiBs
2024-09-09 16:04:12 [Parachain] 🏁 Memory score: 6.01 GiBs
2024-09-09 16:04:12 [Parachain] 🏁 Disk score (seq. writes): 924.06 MiBs
2024-09-09 16:04:12 [Parachain] 🏁 Disk score (rand. writes): 81.53 MiBs
2024-09-09 16:04:12 [Relaychain] discovered: 12D3KooWJAfW5JEbC4SHCVh8Wdi76optU6gz6iGzqLy9UQmChmzB /ip4/144.91.71.235/tcp/30335
2024-09-09 16:04:12 [Parachain] discovered: 12D3KooWFVR3LKVNwNE12ocyquY5eZNsWe3jSCTLxpvFxZgekaWo /ip4/144.91.71.235/tcp/30334/ws
2024-09-09 16:04:13 [Parachain] 🔍 Discovered new external address for our node: /ip4/144.91.71.235/tcp/30335/p2p/12D3KooWJAfW5JEbC4SHCVh8Wdi76optU6gz6iGzqLy9U
QmChmzB
2024-09-09 16:04:13 [Relaychain] 🔍 Discovered new external address for our node: /ip4/144.91.71.235/tcp/30334/ws/p2p/12D3KooWFVR3LKVNwNE12ocyquY5eZNsWe3jSCTLx
pvFxZgekaWo
2024-09-09 16:04:13 [Relaychain] Sending fatal alert BadCertificate
2024-09-09 16:04:14 [Relaychain] Sending fatal alert BadCertificate
2024-09-09 16:04:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2a02:c207:3012:4713::1/tcp/30334/ws/p2p/12D3KooWFVR3LKVNwNE12ocyquY5eZNs
We3jSCTLxpvFxZgekaWo
2024-09-09 16:04:17 [Relaychain] ⏩ Warping, Downloading finality proofs, 0.00 Mib (4 peers), best: #0 (0x77af…764f), finalized #0 (0x77af…764f), ⬇ 101.1kiB/s
⬆ 35.0kiB/s
2024-09-09 16:04:17 [Parachain] 💤 Idle (0 peers), best: #0 (0x219e…ffe2), finalized #0 (0x219e…ffe2), ⬇ 1.1kiB/s ⬆ 1.0kiB/s

al3mart commented 2 months ago

Hey!

Another user has let me know that the guide is missing pointing to change the relay_chain field of the spec with the correct value.

In yours I can see it is set to Kusama -> "relay_chain": "kusama", that should be: "relay_chain": "paseo",. I don't think that would be the only cause. I would advise to purge the parachain db and launch the collators again with the correct spec.

We might need to re-register the wasm and state of the proper spec :+1:

gianfra-t commented 2 months ago

Thanks for looking into this!

We modified the relay_chain field but we see the exact same issue Parachain] Could not find the header of the genesis block..... Regarding the state with this new spec, the modification of the field does not affect the state we got before.

But, we realized we did not removed the key 0x45323df7cc47150b3930e2666b0aa313a2bca190d36bd834cc73a38fc213ecbd as it says hinted on the docs (my fault, I thought it was an optional step for particular cases).

Upon removal of that key we do get a different state, which I upload here: parachain-genesis-head-for-paseo.txt.

As well as the WASM:

parachain-genesis-wasm-for-paseo.txt

We tried running with this new spec and same log error. Only this time the genesis state reflects that I'm sharing right now.

Do you think this could explain (and solve) this?

al3mart commented 2 months ago

I have set the new head and wasm, please run the collators with the new spec now :+1:

Don't forget to purge the parachain db.

gianfra-t commented 2 months ago

Sadly it didn't work. Same missing block error:

2024-09-09 11:47:36 [Parachain] Could not find the header of the genesis block in the database! block_hash=0x2b5696b3842c4f144175180613feabee6940c5471d4871e94791023c94c66338

Sorry I didn't answer yesterday, we were discussing a bit internally to see if we could find the cause.

We realize that given the registered genesis head: 0x00000000000000000000000000000000000000000000000000000000000000000013cb6f9ea24664610edda75de68c7244358c156b8fd9449590de5efb78df9a6c03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400

it's hash, the hash of the genesis block the node is looking for in the database is: 0x2b5696b3842c4f144175180613feabee6940c5471d4871e94791023c94c66338

Which corresponds with the error log message. Now, as you can see on the log message I shared in the comment above, the first block the parachain generates is actually:

[Parachain] 💤 Idle (0 peers), best: #0 (0x21ae…77e5), finalized #0 (0x21ae…77e5)

Any idea what could be the cause this mismatch when starting the node?

al3mart commented 2 months ago

Most likely the genesis state that is registered in the relay is wrong. Otherwise a parachain db prune might just do the trick.

I'd suggest to export the genesis head and double check that is the one you are expecting and that in fact is not the same than the one that has been registered in the relay. In case the registered head matches, the db most likely maintains state from an old run of the chain with a different genesis, and that's were the error might be coming from. So prunning the db and restarting the service should do it.

gianfra-t commented 2 months ago

We finally managed to get the parachain to produce blocks. We think it didn't have to do with the pruning but with our initial state itself. I'll leave here the findings in case they are helpful.

First issue, mismatch between genesis head and state root.

This is the initial issue we had, where we got:

2024-09-09 11:47:36 [Parachain] Could not find the header of the genesis block in the database! block_hash

After realizing that the hash of the registered genesis state head is used to find the first block (as it should correspond to the genesis block hash), we found a mismatch between the head generated by the command and what the node actually uses to create the first block.

When exploring the first block created we saw the following log:

2024-09-09 16:04:04 [Parachain] 🔨 Initializing Genesis block/state (state: 0xec8e…f88c, header-hash: 0x219e…ffe2)

Keeping in mind the invariant that 0x000000000000000000000000000000000000000000000000000000000000000000${genesisStateRoot}03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400, we realized that the state root used to create the first block did not match that of the head created with export-genesis-state.

In this case, we would expect the output of export-genesis-state to be: 0x000000000000000000000000000000000000000000000000000000000000000000{ec8e…f88c}03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400, yet it was something different.

We discovered that when our genesis spec file contains childrenDefault key - value pairs, the calculated genesis state head is not equal to the one used to produce the first block, more precisely, the state root.

After we removed childredDefault we got a match between what is calculated using export-genesis-state (which goes into the relay) and the actual head and state root our parachain uses.

Second issue, DmqHead

Then, we encountered the following issue when collating:

2024-09-16 12:03:36 [Parachain] panicked at .../pallets/parachain-system/src/lib.rs:861:9: assertion `left == right` failed left: 0x91cd3332041afe9540945179c6518a82f3261632a47bb7acb34264be14412def right: 0x0000000000000000000000000000000000000000000000000000000000000000

We saw that parachainSystem.lastDmqMqcHead() does not correspond to what the expected dmq_head coming from the relaychain.

Just to be on the safe side, we decided to reset all the states of the following pallets (not comprehensive, but probably the most important):

Parachain System 
Timestamp 
Parachain Info
Dmp Queue
Xcmp Queue
Randomness flip

At the very least, we presume that lastDmqMqcHead was required to be reset to solve this second issue.

In our case, we did this with a script that:

Iterates the initial string prefixes of the pallet(s) we want to reset.
Either
- Replace with the value that exists on an empty spec of our chain. (except for storage version!)
- Delete the value if it is not in the empty spec.

With this modifications, we were able to re calculate the genesis head, add it to the relay and actively produce blocks!

ebma commented 2 months ago

We fully completed the migration and our parachain is producing blocks reliably, see here. Thanks for your assistance in the process @al3mart! Our parachain (Foucoco) is still listed in the "TEST ROCOCO & PARACHAINS" list in the sidebar of the polkadot.js explorer. Do you know where we need to reach out to or where we can submit a PR to move the listing to "TEST PASEO & PARACHAINS" instead?

Besides that, I guess this ticket is done and can be closed.

al3mart commented 1 month ago

Awesome!! That's great to hear.

You can include your parachain in the polkadot js application opening a PR like this one - https://github.com/polkadot-js/apps/pull/10934

paseo-network / support