Update to SDK `stable2409-1`

pandres95 commented 1 month ago

Closes #457 Closes #469

Updating the runtimes to SDK version stable2409-1. CHANGELOG mentions all relevant changes for UI and Walletbuilders.

Checklist

[x] Update SDK to stable2409
[x] Update configuration for relay chains.
[x] Update configuration for system parachains.
[x] Update configuration for emulated chains.
[x] Update configuration for Encointer (see Help wanted).
[x] Update CHANGELOG.
[x] Update SDK to stable2409-1

Dependencies

This PR depends on certain PRs to be merged before it can properly work.

[x] https://github.com/polkadot-fellows/runtimes/pull/472

Impacts

The success of this PR impacts directly on the feasibility of executing certain issues, or closing certain PRs.

Help wanted

Initially, I request the help of:

@franciscoaguirre to help me correcting the potential conflicts that may arise when foreign assets change to XCMv4.
@acatangiu to help me setting up the Bridge Hubs, as well as adding the respective transition layer (a.k.a. preparing the migrations from fixed Lanes to LegacyLane-based lanes storage).
@clangenb to help me upgrading Encointer.
@Szegoo to check the parameters that changed in Coretime.

clangenb commented 1 month ago

I can probably squeeze it in later this week. 👍

pandres95 commented 1 month ago

Opened for review with a small caveat: tests will keep failing, until #472 is merged and can be merged back to this branch.

bkontur commented 1 month ago

@pandres95 if you have anything related to the Bridges, just tell me, at least I can see that migrations are missing here:

    pallet_bridge_messages::migration::v1::MigrationToV1<
    bridge_to_kusama/polkadot_config::migration::StaticToDynamicLanes,  // check this for bridge_to_westend_config::migration::StaticToDynamicLanes
    frame_support::migrations::RemoveStorage<  // check bridge-hub-westend for removing BridgeRococoMessagesPalletName
        Bridge***MessagesPalletName,
        OutboundLanesCongestedSignalsKey,
        RocksDbWeight,
    >,
    pallet_bridge_relayers::migration::v1::MigrationToV1<Runtime, ()>,

and also I can help with fixing (integration) tests

clangenb commented 1 month ago

Is it reasonable to assume that we are going to use 2409-1? I would update encointer directly to that version then.

acatangiu commented 1 month ago

@acatangiu to help me setting up the Bridge Hubs, as well as adding the respective transition layer (a.k.a. preparing the migrations from fixed Lanes to LegacyLane-based lanes storage).

I suggest we keep this SDK as light as possible:

@bkontur will help with migrating existing legacy lane to work with the new SDK deps,
we will follow-up with a separate PR to configure and deploy the new permissionless lanes functionality (split it out of this PR).

pandres95 commented 1 month ago

@Szegoo @seadanda I just applied the changes you suggested. @bkontur I would love to have some help on the integration tests. As of now, I think some tests will fail due to the issue of not having foreign assets with xcmv4 format in asset hubs (see #472). Once that's done, we can begin with those integration tests.

clangenb commented 1 month ago

@pandres95 did you see that I opened https://github.com/pandres95/runtimes/pull/1 into your PR?

pandres95 commented 1 month ago

Failed to update to stable2409-1 (see commit f94520c) due to new versions of the updated crates not yet published (see polkadot-sdk#493).

github-actions[bot] commented 1 month ago

Command help:

``` usage: /cmd bench [-h] [--continue-on-fail] [--quiet] [--clean] [--runtime [{polkadot,kusama,glutton-kusama,asset-hub-kusama,asset-hub-polkadot,bridge-hub-kusama,bridge-hub-polkadot,collectives-polkadot,coretime-kusama,coretime-polkadot,people-kusama,people-polkadot,encointer-kusama} ...]] [--pallet [PALLET ...]] options: -h, --help show this help message and exit --continue-on-fail Won't exit(1) on failed command and continue with next steps. Helpful when you want to push at least successful pallets, and then run failed ones separately --quiet Won't print start/end/failed messages in Pull Request --clean Clean up the previous bot's & author's comments in Pull Request which triggered /cmd --runtime [{polkadot,kusama,glutton-kusama,asset-hub-kusama,asset-hub-polkadot,bridge-hub-kusama,bridge-hub-polkadot,collectives-polkadot,coretime-kusama,coretime-polkadot,people-kusama,people-polkadot,encointer-kusama} ...] Runtime(s) space separated --pallet [PALLET ...] Pallet(s) space separated **Examples**: > runs all benchmarks /cmd bench > runs benchmarks for pallet_balances and pallet_multisig for all runtimes which have these pallets > --quiet makes it to output nothing to PR but reactions /cmd bench --pallet pallet_balances pallet_xcm_benchmarks::generic --quiet > runs bench for all pallets for polkadot runtime and continues even if some benchmarks fail /cmd bench --runtime polkadot --continue-on-fail > does not output anything and cleans up the previous bot's & author command triggering comments in PR /cmd bench --runtime polkadot kusama --pallet pallet_balances pallet_multisig --quiet --clean ```

pandres95 commented 1 month ago

@acatangiu I found that in kah integration tests, the chain 11155111 is being used as the origin chain for wETH. So, I made this change: https://github.com/pandres95/runtimes/blob/f8adb35e0efda35c96b5e3913e60f7f201aecfcb/system-parachains/asset-hubs/asset-hub-kusama/src/xcm_config.rs#L516-L521

Is this alright? Or should we change that value in the tests and point to the chain 1?

pandres95 commented 1 month ago

/cmd bench --runtime bridge-hub-kusama --continue-on-fail

github-actions[bot] commented 1 month ago

Command "--runtime bridge-hub-kusama --continue-on-fail" has started 🚀 See logs here

github-actions[bot] commented 1 month ago

Command "--runtime bridge-hub-kusama --continue-on-fail" has failed ❌! See logs here

acatangiu commented 4 weeks ago

@acatangiu I found that in kah integration tests, the chain 11155111 is being used as the origin chain for wETH. So, I made this change: https://github.com/pandres95/runtimes/blob/f8adb35e0efda35c96b5e3913e60f7f201aecfcb/system-parachains/asset-hubs/asset-hub-kusama/src/xcm_config.rs#L516-L521

Is this alright? Or should we change that value in the tests and point to the chain 1?

The Rococo/Westend testnets are bridged to Ethereum Sepolia testnet which is using chain_id = 11155111.

But Ethereum chain_id should be 1 throughout this repo, as that's Ethereum Mainnet which Polkadot is bridged to.

claravanstaden commented 4 weeks ago

@pandres95 I added the Snowbridge Polkadot-native token transfer to Ethereum integration tests here: https://github.com/pandres95/runtimes/pull/2

@acatangiu we need to set the ReserveAssetDeposited weight in PBH to something realistic for PNA to work.

acatangiu commented 4 weeks ago

@acatangiu we need to set the ReserveAssetDeposited weight in PBH to something realistic for PNA to work.

why? where would BridgeHub receive/execute a ReserveAssetDeposited from? BH only handles DOT which cannot be transferred into BH as a reserve transfer with a remote reserve.

claravanstaden commented 4 weeks ago

why? where would BridgeHub receive/execute a ReserveAssetDeposited from? BH only handles DOT which cannot be transferred into BH as a reserve transfer with a remote reserve.

Our EthereumBlobExporter expects a message that contains ReserveAssetDeposited. The exporter then parses the xcm payload to send a command to Ethereum. So it does not actually execute ReserveAssetDeposited with the XCM executor, but I think the message is still weighed per instruction.

acatangiu commented 3 weeks ago

Our EthereumBlobExporter expects a message that contains ReserveAssetDeposited. The exporter then parses the xcm payload to send a command to Ethereum. So it does not actually execute ReserveAssetDeposited with the XCM executor, but I think the message is still weighed per instruction.

AFAIK, the XCM is weighed as a first step of execution. If BridgeHub is not executing the XCM, why would it weigh it?

I was expecting the XCM is captured, parsed and consumed by the Snowbridge exporter before being weighed or executed. ~Ideally it should be captured even before trying to go through the BH XCM barriers, but I'm not sure about that, needs checking.~ L.E.: barriers are used during execution, so they should also not be a problem.

claravanstaden commented 3 weeks ago

@acatangiu you are right, its not the weight that is a problem. It is related to withdrawing the fee asset it seems:

2024-10-29T10:24:55.407179Z TRACE xcm::bridge-hub-router: Router with bridged_network_id Kusama does not support bridging to network Ethereum { chain_id: 1 }!    
2024-10-29T10:24:55.407185Z TRACE xcm::bridge-hub-router: validate - ViaBridgeHubExporter - error: NotApplicable    
2024-10-29T10:24:55.407318Z TRACE xcm::pallet_xcm::note_unknown_version: XCM version is unknown for destination: Location { parents: 1, interior: X1([Parachain(1002)]) }    
2024-10-29T10:24:55.407504Z TRACE xcm::contains: AllSiblingSystemParachains location: Location { parents: 0, interior: X1([AccountId32 { network: Some(Polkadot), id: [212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125] }]) }    
2024-10-29T10:24:55.408142Z TRACE xcm::fungible_adapter: withdraw_asset what: Asset { id: AssetId(Location { parents: 1, interior: Here }), fun: Fungible(340282366920938463463374607431768211455) }, who: Location { parents: 0, interior: X1([AccountId32 { network: Some(Polkadot), id: [212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125] }]) }

claravanstaden commented 3 weeks ago

@pandres95 @acatangiu test fixes over here: https://github.com/pandres95/runtimes/pull/4 Thanks @alistair-singh for the BridgeHubEthereumBaseFee tip, you were right. 🏆

pandres95 commented 3 weeks ago

@acatangiu @bkontur any clue about this error?

https://github.com/polkadot-fellows/runtimes/actions/runs/11574125246/job/32217719672#step:11:244

bkontur commented 3 weeks ago

@acatangiu @bkontur any clue about this error?

https://github.com/polkadot-fellows/runtimes/actions/runs/11574125246/job/32217719672#step:11:244

I think it should be related to the frame-omni-bencher bug, let me check that and try it tomorrow

pandres95 commented 3 weeks ago

Inviting @Szegoo @seadanda and @bkontur to give it a second review.

franciscoaguirre commented 2 weeks ago

What's the state of this? Only needs reviews?

pandres95 commented 2 weeks ago

Only reviews @franciscoaguirre. There are some "failing" migrations, but the failure is related to the spec version. Once we bump, those errors should disappear.

pandres95 commented 2 weeks ago

/cmd help

github-actions[bot] commented 2 weeks ago

Command "help" has started 🚀 See logs here

github-actions[bot] commented 2 weeks ago

Command "help" has failed ❌! See logs here

pandres95 commented 2 weeks ago

/cmd --help

github-actions[bot] commented 2 weeks ago

Command help:

``` usage: /cmd [--help] {bench,fmt} ... A command runner for polkadot runtimes repo positional arguments: {bench,fmt} a command to run bench Runs benchmarks fmt Formats code options: --help help for help if you need some help ### Command 'bench' usage: /cmd bench [-h] [--continue-on-fail] [--quiet] [--clean] [--runtime [{polkadot,kusama,glutton-kusama,asset-hub-kusama,asset-hub-polkadot,bridge-hub-kusama,bridge-hub-polkadot,collectives-polkadot,coretime-kusama,coretime-polkadot,people-kusama,people-polkadot,encointer-kusama} ...]] [--pallet [PALLET ...]] options: -h, --help show this help message and exit --continue-on-fail Won't exit(1) on failed command and continue with next steps. Helpful when you want to push at least successful pallets, and then run failed ones separately --quiet Won't print start/end/failed messages in Pull Request --clean Clean up the previous bot's & author's comments in Pull Request which triggered /cmd --runtime [{polkadot,kusama,glutton-kusama,asset-hub-kusama,asset-hub-polkadot,bridge-hub-kusama,bridge-hub-polkadot,collectives-polkadot,coretime-kusama,coretime-polkadot,people-kusama,people-polkadot,encointer-kusama} ...] Runtime(s) space separated --pallet [PALLET ...] Pallet(s) space separated **Examples**: > runs all benchmarks /cmd bench > runs benchmarks for pallet_balances and pallet_multisig for all runtimes which have these pallets > --quiet makes it to output nothing to PR but reactions /cmd bench --pallet pallet_balances pallet_xcm_benchmarks::generic --quiet > runs bench for all pallets for polkadot runtime and continues even if some benchmarks fail /cmd bench --runtime polkadot --continue-on-fail > does not output anything and cleans up the previous bot's & author command triggering comments in PR /cmd bench --runtime polkadot kusama --pallet pallet_balances pallet_multisig --quiet --clean ### Command 'fmt' usage: /cmd fmt [-h] [--continue-on-fail] [--quiet] [--clean] options: -h, --help show this help message and exit --continue-on-fail Won't exit(1) on failed command and continue with next steps. Helpful when you want to push at least successful pallets, and then run failed ones separately --quiet Won't print start/end/failed messages in Pull Request --clean Clean up the previous bot's & author's comments in Pull Request which triggered /cmd ```

pandres95 commented 2 weeks ago

@SBalaguer @anaelleltd just to mention this PR aims to update the Polkadot SDK version of the runtimes to the latest stable one (stable2409-1).

anaelleltd commented 6 days ago

To reviewers: Please take care to double check this update carefully and advise if there are potential breaking changes or disruptions for ecosystem builders. If any, please ping @anaelleltd or @SBalaguer in your detailed commentary so that we can notify ecosystem teams with the critical information.

polkadot-fellows / runtimes