Slot request for Zeitgeist

sea212 commented 1 year ago

You completed the Parachain Tutorials for your parachain and tested locally?

[X] Yes, we have successfully tested our chain on rococo-local.

Team Name

Zeitgeist

Parachain Name

Zeitgeist

ParaID

2101

Is this for a short or long term slot?

Long-term: we have a matching paraID in use on Polkadot or Kusama.

Anything else we should know?

We want to test XCM integrations with other parachains.

Register your Rococo parachain details for `polkadot-js/apps`

[x] Polkadot-js Apps UI PR completed
- Available here: https://github.com/polkadot-js/apps/pull/8231

al3mart commented 1 year ago

registering parathread

al3mart commented 1 year ago

I have assigned your para 30 slots, as the long-term paras are capped atm, we are performing a runtime upgrade shortly to bump this value. This should get you going until the new runtime is enacted.

sea212 commented 1 year ago

@al3mart The genesis state and gensis wasm provided here were tested on rococo-local v0.9.26 and v0.9.30 and the parachain successfully produced blocks. However, on the live network it cannot produce the first block. What could be the reason for that?

al3mart commented 1 year ago

Looks at this line Oct 29 17:36:25 battery-station-xcm-test-servers zeitgeist[476963]: Thread 'tokio-runtime-worker' panicked at 'Invalid messaging state in relay chain state proof: DmqMqcHead(Proof)', /home/sea212/.cargo/git/checkouts/cumulus-a92b011ec2b4c70b/82c9c1f/pallets/parachain-system/src/lib.rs:352 It looks like there is some mismatch between the mqcHeads state. The storage on the relay side seems clean, as we are doing a fresh onboarding for this para. Could you check if on your side you might have populated some storage related to this ?

al3mart commented 1 year ago

Other option, but I guess you have already checked this. Are the wasm and state submitted in the issue the correct ones ? :thinking:

Chralt98 commented 1 year ago

Other option, but I guess you have already checked this. Are the wasm and state submitted in the issue the correct ones ?

Thank you so much. We checked them and they are correct.

Apart from that:

When I query the rococo relay chain for storage -> dmp -> downwardMessageQueueHeads(u32): H256 for parachain id 2101 (or 2050) I get no hash back. That's what you meant above with The storage on the relay side seems clean, as we are doing a fresh onboarding for this para., right? Maybe the call sudo_queue_downward_xcm is required? Or where does the setup of the downward message queue head happen?

fn read_entry<T, B>(backend: &B, key: &[u8], fallback: Option<T>) -> Result<T, ReadEntryErr>
where
    T: Decode,
    B: Backend<HashFor<relay_chain::Block>>,
{
        // WHERE DOES THIS BACKEND GET WRITTEN WITH dmq_mqc_head(self.para_id) ?
    backend
        .storage(key)
                 // IT FAILS TO READ THE KEY (DmqMqcHead(Proof))
        .map_err(|_| ReadEntryErr::Proof)?
.
.
.
let dmq_mqc_head: relay_chain::Hash = read_entry(
    &self.trie_backend,
    &relay_chain::well_known_keys::dmq_mqc_head(self.para_id),
    Some(Default::default()),
)
.map_err(Error::DmqMqcHead)?;

Inside the trie_backend there isn't the key well_known_keys::dmq_mqc_head(self.para_id), so that's why I expected to get a value when I queried the storage for downwardMessageQueueHeads on the rococo relaychain. The para_id is in our collator node from parachain_info. You can query it on the Battery Station: It is 2050 (parachainInfo -> parachainId) there instead of 2101. This is a misconfiguration and needs to be solved. But we can't verify if this is the cause of the problem, because we did start a local rococo chain with the collator and it produced blocks (with the same genesis configuration as here). Does the local rococo setup somehow fill the trie_backend with the dmq_mqc_head (downwardMessageQueueHead) (with the specified paraId from here) ?

I think this test from polkadot github was interesting:

#[test]
fn verify_dmq_mqc_head_is_externally_accessible() {
    use hex_literal::hex;
    use primitives::v2::well_known_keys;

    let a = ParaId::from(2020);

    new_test_ext(default_genesis_config()).execute_with(|| {
        let head = sp_io::storage::get(&well_known_keys::dmq_mqc_head(a));
        assert_eq!(head, None);

        queue_downward_message(a, vec![1, 2, 3]).unwrap();

        let head = sp_io::storage::get(&well_known_keys::dmq_mqc_head(a));
        assert_eq!(
            head,
            Some(hex!["434f8579a2297dfea851bf6be33093c83a78b655a53ae141a7894494c0010589"].to_vec())
        );
    });
}

sea212 commented 1 year ago

Looks at this line Oct 29 17:36:25 battery-station-xcm-test-servers zeitgeist[476963]: Thread 'tokio-runtime-worker' panicked at 'Invalid messaging state in relay chain state proof: DmqMqcHead(Proof)', /home/sea212/.cargo/git/checkouts/cumulus-a92b011ec2b4c70b/82c9c1f/pallets/parachain-system/src/lib.rs:352 It looks like there is some mismatch between the mqcHeads state. The storage on the relay side seems clean, as we are doing a fresh onboarding for this para. Could you check if on your side you might have populated some storage related to this ?

Thanks for the fast response @al3mart. What exactly do we have to check? parachainSystem.lastDmqMqcHead and parachainSystem.lastHrmpMqcHeads return nothing. Are there any obvious reasons why it could work on a local rococo instance (using the same version as the live instance) but not on the live one? Are there any requirements in regards to the Cumulus/Polkadot dependency version within the parachain runtime? The genesis wasm contains a runtime that's based on v0.9.19, we could provide another at v0.9.26 shortly.

al3mart commented 1 year ago

dmq_mqc_head only gets populated when messages are going into the queue. Fix your paraId for the parachain_info module, and let's see the behaviour of your runtime based on v0.9.26

sea212 commented 1 year ago

@al3mart Can you please update the genesis code and head to the following:

State: bs_parachain_rococo_genesis_state.zip
Wasm: bs_parachain_rococo_genesis_wasm.zip

We also need a lease extension.

For the sake of completeness, here is also the chainspec: bs_parachain_rococo.zip

al3mart commented 1 year ago

All of the above have been addressed. Onboarding happening in 1h aprox from now

sea212 commented 1 year ago

The chain produces blocks now, thanks for the quick update @al3mart .

sea212 commented 1 year ago

We experienced a chain stall which seems to be caused by a XCM message that arrived before we deployed our proper XCM configuration. The collators still produce blocks and forward them to the assigned validators, however those blocks never seem to be accepted. I hate to ask once more; Could please reset the state and set a new genesis state and wasm @al3mart ? is there a default procedure for such requests already in place?

al3mart commented 1 year ago

New state and wasm set.

Actually, by sending comments to this very same issue you are making the process easy to follow. Hopefully in the future you wouldn't need to interact with a person, but for the time being this is the more tidy and organized way of managing such requests.

sea212 commented 1 year ago

Thanks for the quick reset @al3mart. The node panics because its local (empty) dmqHead is 0x00000..., whereas the one on the relay chain dmp.downwardMessageQueueHeads(2101) is 0x5a452a072cc75379842f9b5319a7f9f6436c6e0399f26596c49ff6b6d2fe5f3a . Can you reset this on the relay chain or is that something that we have to hardcode in the genesis storage?

al3mart commented 1 year ago

done, thanks for the heads up

NM2146 commented 1 year ago

Hello @sea212, Can you please share with us your Parachain Manager Account. A Rococo account that would be set as the manager for this parachain. You might have already set one if you have registered the parachain on your own, please use the same account. This will allow you to have better control and flexibility in managing your Para.

sea212 commented 9 months ago

@al3mart @NM2146 Please set 5HGNBoNqrDayk7WdHX7eM4LApWHhW8xYBn4JnztYSwFbcbaN as the manager and also unlock the parachain (2101). Quick action will be much appreciated, as we have to overwrite the head and state right now.

AlexD10S commented 9 months ago

Parachain unlocked and the new manager account has been set to 5HGNBoNqrDayk7WdHX7eM4LApWHhW8xYBn4JnztYSwFbcbaN

sea212 commented 9 months ago

@al3mart @NM2146 @AlexD10S We tried to revert our bricked parachain on Rococo by specifying badBlocks in our chainspec and using the ability of the manager to set the head to a previous state and (unfortunately only) schedule a code upgrade. It seems that reverting will not work this way, as we try to reset the parachain to a prior runtime.

We apparently need help of sudo here, please reset any scheduled code upgrade for parachain 2,101 and force set the code for parachain 2,101 to https://github.com/zeitgeistpm/zeitgeist/releases/download/v0.3.11/battery_station_runtime.compact.compressed.wasm

AlexD10S commented 9 months ago

If I query scheduler -> agenda(2101) there is nothing scheduled for this parachain. Am I missing something? I can force set the code with the new one

sea212 commented 9 months ago

I think Rococo sent green lights to the parachain to apply the upgrade, but the parachain cannot handle it.

Querying paras.futureCodeUpgrade(2101) results in 7,245,741, which was already in the past. paras.upgradeGoAheadSignal(2101) is set to GoAhead and paras.futureCodeHash(2101) is set to 0xfe1aab96ec2a438d6355f08b2fc24b9af2e68b2fa937f7980f971e0f5ad92ef0.

AlexD10S commented 9 months ago

I force a new code upgrade with force_scheduled_code_upgrade(): https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo-rpc.polkadot.io#/explorer/query/0xd17c45f57fdb457bf584e56578122a9cc0a1541ceaebea4831f74dc174740c80

sea212 commented 9 months ago

Thank you, but I doubt that this will work, as the runtime provided is the same that's used for the given head that was set by us earlier. I think the approach to take here is to reset the future code hash, reset the go ahead signal so it is not sent and finally forceSetCode to the provided runtime.

AlexD10S commented 9 months ago

Sorry, I understand now. Let me do a bit more research because I don't think reset reset the future code hash and reset the go ahead signal is trivial, there are no extrinsics for that (I believe), so we might set manually the storage.

AlexD10S commented 9 months ago

All these fields reset and forceSetCode with the new code executed!

sea212 commented 7 months ago

@AlexD10S The slot leases of Battery Station (2101) expired, but the production equivalent is still live on Polkadot (Zeitgeist, 2092). I assume it was coupled to the Kusama slot leases previously, but we Zeitgeist was migrated to Polkadot early this year. Please update the slot leases on Rococo accordingly.

al3mart commented 7 months ago

Hey @sea212, thanks for the heads up, your chain should be up and running again in 1h from now :)

sea212 commented 2 months ago

Hey @al3mart , could you please open a HRMP channel between AssetHub-Rococo and Zeitgeist (2101)?

Btw. I tried to do it myself, but it seems to me I have to use the Root track on AssetHub which requires a 100k deposit. What is the correct way for parachain teams to open channels to system chains on Rococo?

paritytech / subport