search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.63k stars 571 forks source link

Pubsub mechanism for Parachains #606

Open bkchr opened 1 year ago

bkchr commented 1 year ago

When parachains need to exchange information, they currently rely on sending XCM messages to each other or resorting to an off-chain protocol to obtain a storage proof from another chain. However, these solutions are complex and expensive, particularly when data should be disseminated amongst a large group of parachains, such as the price of DOT.

A potential solution would be the introduction of a publish-subscribe (pub-sub) mechanism. This would enable parachains to subscribe to (para_id, key) information within their runtime. Collators would then be responsible for integrating this information into the runtime during block creation. The information would be published by parachains as an output of their validation process, necessitating changes to the ValidationResult as specified in issue paritytech/polkadot-sdk#645.

The (k, v) information would be stored in the relay chain state, and it would likely be most efficient to maintain a separate child trie for each para_id. This would allow a collator to notify the parachain runtime when a change occurs in one of the subscribed items.

However, the parachain runtime cannot fully trust the collator to include the data. Therefore, we only want to include the data when changes occur to conserve space. In such cases, the collator could include the child trie root (as part of the relay chain proof), which would be sufficient to prove that no changes occurred or provide proof of the child trie with the changed data.

CC @gavofyork

vstam1 commented 8 months ago

Hi @bkchr, I was wondering if you have an update on the pubsub mechanism? I think a lot of parachains would really benefit from this feature, and I was wondering if there is a way we can contribute? Thanks!

JuaniRios commented 8 months ago

I also could really use this feature

wischli commented 8 months ago

Same here!

bkchr commented 8 months ago

Yeah we have talked a little bit "off github" about this. The current plan is the following:

  1. Introduce a new XCM message to publish certain information. This will be send from the parachain to the relay chain.
  2. Then have the parachain read the relay chain state and put it into the ValidationData inherent. (I'm working on this already, but on some kind of optimized version.)

If you need this, could you may look into 1? Providing 2 in some "dirty" way should be possible quite easily.

vstam1 commented 8 months ago

Thanks for your quick response. I was wondering if there is another way of publishing the information in step 1 without XCM? Or at least without a new XCM instruction. This new instruction will then probably be introduced in XCM v5 and that can still take months. Might we be able to use the Transact instruction to publish the information?

bkchr commented 7 months ago

@franciscoaguirre what is the status on XCMv4? Still possible to sneak in some new instruction?

franciscoaguirre commented 7 months ago

We can. We wanted to get it out as soon as possible but the PR still doesn't have enough reviews. A new instruction wouldn't hurt

bkchr commented 7 months ago

Nice! Could we add some kind of instruction like this:

Publish { data: Vec<(Key, Value)>

While Key and Value just being Vec<u8>.

franciscoaguirre commented 7 months ago

We can work on the implementation here: https://github.com/paritytech/polkadot-sdk/pull/2359, it's pointing to the v4 branch.

In the meantime, I created an RFC for this to discuss how it should work: https://github.com/paritytech/xcm-format/pull/48 I feel like this could be a more generic instruction like Store that could potentially be used for storage parachains in the future. I see many people want this, please go and comment in there to make a nice new addition to the language :)

franciscoaguirre commented 3 months ago

Since XCMv4 was just a rename, this'll go in XCMv5. I urge you again to take a look at the RFC and comment. This will only go into v5 if we're all convinced about that RFC.