Parachain can't receive message?

[AurevoirXavier]

          strange... when i sent the message the parachains seemed stalled. Maybe the it was something with RPC... i see all good from my side as well.

Good news is that i managed to open the HRMP channel and also the asset ID 7777 was set as sufficient as requested.

In case something is not working as expected please feel free to re open the issue.

Originally posted by @hbulgarini in https://github.com/paseo-network/support/issues/75#issuecomment-2223461452

[AurevoirXavier]

I believe I'm experiencing the same issue as @hbulgarini.

I can send messages to the relay chain, but when I send messages to the assethub, they are not received.

[hbulgarini]

Hey!

We have detected that the HRMP config values are much smaller than in polkadot.

We will take a look at these numbers and update the max values for your channel so you can try again. This is the first that come into my mind that might cause the messages to not arrive to the AH.

[AurevoirXavier]

Hey!

We have detected that the HRMP config values are much smaller than in polkadot.

We will take a look at these numbers and update the max values for your channel so you can try again. This is the first that come into my mind that might cause the messages to not arrive to the AH.

Great! Once you've done that, let me know. Thanks so much.

[al3mart]

I have closed and send the extrinsics to open the channels again, they should take effect in the coming sessions. Please, try again once they are back open and let us know if it is any different

[AurevoirXavier]

I have closed and send the extrinsics to open the channels again, they should take effect in the coming sessions. Please, try again once they are back open and let us know if it is any different

It looks like still can not send message to AssetHub.

Will try again later.

[hbulgarini]

It looks like still can not send message to AssetHub.

Hey @AurevoirXavier , did you have the chance to try again?

[hbulgarini]

btw, the sibling account of your parachain in the AH had 0 PAS. I suspect this was the main reason for the failure.

I have just transferred 100PAS to the 2105 account in the AH.

[hbulgarini]

I saw some messages coming from your parachain that exceeded the weights, please take into account that Paseo (and its system chains) weights are re-benchmarked to meet the Paseo HW:

https://github.com/paseo-network/paseo-action-submission/blob/main/pas/PAS-7-Hardware_specs.md

PS: I think the original issue has gone as messages are arriving to the AH. Also i confirmed it with another para that has created the foreign asset.

[al3mart]

Please, @AurevoirXavier let us know if now your issue has been fixed so we can close this :)

[jiguantong]

I saw some messages coming from your parachain that exceeded the weights, please take into account that Paseo (and its system chains) weights are re-benchmarked to meet the Paseo HW:

https://github.com/paseo-network/paseo-action-submission/blob/main/pas/PAS-7-Hardware_specs.md

PS: I think the original issue has gone as messages are arriving to the AH. Also i confirmed it with another para that has created the foreign asset.

It still doesn't work. Could you please provide some logs about "messages coming from para2105 to AssetHubPaseo"?

[hbulgarini]

I saw some messages coming from your parachain that exceeded the weights, please take into account that Paseo (and its system chains) weights are re-benchmarked to meet the Paseo HW: https://github.com/paseo-network/paseo-action-submission/blob/main/pas/PAS-7-Hardware_specs.md PS: I think the original issue has gone as messages are arriving to the AH. Also i confirmed it with another para that has created the foreign asset.

It still doesn't work. Could you please provide some logs about "messages coming from para2105 to AssetHubPaseo"?

We need to ask the collator maintainers the logs, however i would recommend trying with chopsticks first as this issue seems to be something specific with XCM and it can be easily reproduced with it.

[jiguantong]

We need to ask the collator maintainers the logs, however i would recommend trying with chopsticks first as this issue seems to be something specific with XCM and it can be easily reproduced with it.

Which chain do you mean by forking via chopsticks? I'm not sure if chopsticks can debug this issue. The XCM message was sent from para(2105) successfully, but was not received in AssetHub.

[hbulgarini]

The messages are arriving to the AH from the para 2105. Yesterday i was able to see that in the events and this is the reason why i shared the weights difference with xavier. Here you have the logs, but debugging flags are not turned on. I have requested to turn on the XCM log so any issue can be detected in the logs:

ahp_20240723.log.zip

With chopsticks you can fork the whole environment and many parachains as you want and enable the XCM interactions. I found this the best way by far to debug XCM issues as the one you are having.

Below you have an example:

npx @acala-network/chopsticks@latest xcm --r=paseo.chopsticks.yml --p=darwinia.yml --p=paseo.assethub.yml

I share the chopsticks config files for paseo and the AH:

AH

endpoint: wss://pas-rpc.stakeworld.io/assethub
mock-signature-host: true
db: ./db.sqlite
runtime-log-level: 5

import-storage:
  System:
    Account:
      -
        -
          - 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY
        - providers: 1
          data:
            free: 1000000000000000
  Assets:
    Account:
      - [[1984, 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY], { balance: 1000000000 }]
      - [[21, 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY], { balance: 1000000000 }]
    Asset: [[[21], { supply: 1000000000 }]]

Paseo

endpoint:
  - wss://paseo-rpc.dwellir.com
mock-signature-host: true
db: ./db.sqlite
runtime-log-level: 5

import-storage:
  Sudo:
    Key: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY # Alice
  System:
    Account:
      -
        -
          - 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY
        - providers: 1
          data:
            free: 1000000000000000

[hbulgarini]

XCM logs are turned on now, you could try again and let me know so i can share the logs again.

[jiguantong]

XCM logs are turned on now, you could try again and let me know so i can share the logs again.

Thanks! I just sent the XCM message. Should be after AssetHub block 903477.

[hbulgarini]

Updated Logs: ahp_202407231030.log.zip

[jiguantong]

Hi @hbulgarini , I used chopsticks to test sending XCM to AssetHub Rococo and AssetHub Paseo, and the results were different. In AH-Rococo, it worked, but in AH-Paseo, it failed. I don't know why origin_conversion is called in AH-Paseo but not in AH-Rococo. And I think origin_conversion should not be called here.

Para(2105) > AH-Paseo

command

npx @acala-network/chopsticks@latest xcm --r=paseo.yml --p=ah-paseo.yml --p=2105.yml

log

              xcm::origin_conversion  TRACE: SovereignSignedViaLocation origin: MultiLocation { parents: 1, interior: X1(Parachain(2105)) }, kind: Superuser
      xcm::origin_conversion  TRACE: RelayChainAsNative origin: MultiLocation { parents: 1, interior: X1(Parachain(2105)) }, kind: Superuser
      xcm::origin_conversion  TRACE: SiblingParachainAsNative origin: MultiLocation { parents: 1, interior: X1(Parachain(2105)) }, kind: Superuser
      xcm::origin_conversion  TRACE: ParentAsSuperuser origin: MultiLocation { parents: 1, interior: X1(Parachain(2105)) }, kind: Superuser
      xcm::origin_conversion  TRACE: SignedAccountId32AsNative origin: MultiLocation { parents: 1, interior: X1(Parachain(2105)) }, kind: Superuser
         xcm::convert_origin  TRACE: could not convert: origin: MultiLocation { parents: 1, interior: X1(Parachain(2105)) }, kind: Superuser
             runtime::system  TRACE: Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 9276276000, proof_size: 7689 } }
             runtime::system  TRACE: Used block length: 3414
        chopsticks::executor  TRACE: Completed BlockBuilder_apply_extrinsic
        chopsticks::executor  TRACE: Calling BlockBuilder_finalize_block

Para(2105) > AH-Rococo

command

npx @acala-network/chopsticks@latest xcm --r=paseo.yml --p=ah-rococo.yml --p=2105.yml

log

runtime::xcmp-queue-migration  DEBUG: Lazy migration finished: item gone
                 xcm::weight  TRACE: WeightInfoBounds message: Xcm([WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }, SetTopic([])])
                xcm::execute  TRACE: origin: Location { parents: 1, interior: X1([Parachain(2105)]) }, message: Xcm([WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }, SetTopic([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1])]), weight_credit: Weight { ref_time: 0, proof_size: 0 }
               xcm::barriers  TRACE: TrailingSetTopicAsId origin: Location { parents: 1, interior: X1([Parachain(2105)]) }, instructions: [WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }, SetTopic([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1])], max_weight: Weight { ref_time: 299908000, proof_size: 7186 }, properties: Properties { weight_credit: Weight { ref_time: 0, proof_size: 0 }, message_id: None }
               xcm::barriers  TRACE: TakeWeightCredit origin: Location { parents: 1, interior: X1([Parachain(2105)]) }, instructions: [WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }], max_weight: Weight { ref_time: 299908000, proof_size: 7186 }, properties: Properties { weight_credit: Weight { ref_time: 0, proof_size: 0 }, message_id: Some([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1]) }
               xcm::barriers  TRACE: AllowKnownQueryResponses origin: Location { parents: 1, interior: X1([Parachain(2105)]) }, instructions: [WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }], max_weight: Weight { ref_time: 299908000, proof_size: 7186 }, properties: Properties { weight_credit: Weight { ref_time: 0, proof_size: 0 }, message_id: Some([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1]) }
               xcm::barriers  TRACE: WithComputedOrigin origin: Location { parents: 1, interior: X1([Parachain(2105)]) }, instructions: [WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }], max_weight: Weight { ref_time: 299908000, proof_size: 7186 }, properties: Properties { weight_credit: Weight { ref_time: 0, proof_size: 0 }, message_id: Some([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1]) }
               xcm::barriers  TRACE: AllowTopLevelPaidExecutionFrom origin: Location { parents: 1, interior: X1([Parachain(2105)]) }, instructions: [WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }])), ClearOrigin, BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Unlimited }, DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }], max_weight: Weight { ref_time: 299908000, proof_size: 7186 }, properties: Properties { weight_credit: Weight { ref_time: 0, proof_size: 0 }, message_id: Some([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1]) }
                xcm::process  TRACE: origin: Some(Location { parents: 1, interior: X1([Parachain(2105)]) }), total_surplus/refunded: Weight { ref_time: 0, proof_size: 0 }/Weight { ref_time: 0, proof_size: 0 }, error_handler_weight: Weight { ref_time: 0, proof_size: 0 }
    xcm::process_instruction  TRACE: === WithdrawAsset(Assets([Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }]))
xcm::ensure_can_subsume_assets  TRACE: worst_case_holding_len: 1, holding_limit: 64
       xcm::fungible_adapter  TRACE: withdraw_asset what: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, who: Location { parents: 1, interior: X1([Parachain(2105)]) }
      xcm::fungibles_adapter  TRACE: withdraw_asset what: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, who: Location { parents: 1, interior: X1([Parachain(2105)]) }
    xcm::process_instruction  TRACE: === ClearOrigin
    xcm::process_instruction  TRACE: === BuyExecution { fees: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(50000000) }, weight_limit: Limited(Weight { ref_time: 299908000, proof_size: 7186 }) }
                 xcm::weight  TRACE: UsingComponents::buy_weight weight: Weight { ref_time: 299908000, proof_size: 7186 }, payment: AssetsInHolding { fungible: {AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }): 50000000}, non_fungible: {} }, context: XcmContext { origin: None, message_id: [84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1], topic: None }
                 xcm::weight  TRACE: SwapFirstAssetTrader::buy_weight weight: Weight { ref_time: 299908000, proof_size: 7186 }, payment: AssetsInHolding { fungible: {AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }): 50000000}, non_fungible: {} }
                 xcm::weight  TRACE: TakeFirstAssetTrader::buy_weight weight: Weight { ref_time: 299908000, proof_size: 7186 }, payment: AssetsInHolding { fungible: {AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }): 50000000}, non_fungible: {} }, context: XcmContext { origin: None, message_id: [84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1], topic: None }
    xcm::process_instruction  TRACE: === DepositAsset { assets: Wild(AllCounted(1)), beneficiary: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) } }
       xcm::fungible_adapter  TRACE: deposit_asset what: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(42814000) }, who: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) }
      xcm::fungibles_adapter  TRACE: deposit_asset what: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(42814000) }, who: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [220, 255, 18, 25, 18, 22, 135, 57, 19, 83, 177, 126, 121, 139, 16, 232, 127, 110, 87, 139, 42, 1, 224, 50, 55, 95, 47, 20, 160, 113, 43, 87] }]) }
    xcm::process_instruction  TRACE: === SetTopic([84, 197, 88, 116, 29, 65, 66, 156, 67, 243, 189, 10, 120, 19, 252, 126, 217, 115, 95, 139, 53, 225, 75, 156, 194, 30, 105, 193, 109, 165, 1, 1])
                xcm::execute  TRACE: result: Ok(())
         xcm::refund_surplus  TRACE: total_surplus: Weight { ref_time: 0, proof_size: 0 }, total_refunded: Weight { ref_time: 0, proof_size: 0 }, current_surplus: Weight { ref_time: 0, proof_size: 0 }
         xcm::refund_surplus  TRACE: total_refunded: Weight { ref_time: 0, proof_size: 0 }
      xcm::fungibles_adapter  TRACE: deposit_asset what: Asset { id: AssetId(Location { parents: 0, interior: X2([PalletInstance(50), GeneralIndex(7777)]) }), fun: Fungible(7186000) }, who: Location { parents: 0, interior: X1([AccountId32 { network: None, id: [156, 229, 116, 30, 226, 241, 172, 59, 222, 219, 222, 159, 51, 57, 4, 143, 77, 162, 203, 136, 221, 243, 58, 9, 119, 250, 11, 76, 248, 110, 41, 72] }]) }
        xcm::process-message  TRACE: XCM message execution complete, used weight: Weight(ref_time: 299908000, proof_size: 7186)

[hbulgarini]

Hello @jiguantong,

I would recommend you to take a look at all the different potential issues that might come up when using XCM in general. From our side, the Paseo AH is an exact copy of the Polkadot AH, so from the configuration perspective we are pretty confident that everything is correct. In fact, another team was able to register a foreign asset in the Paseo AH executing a transact. It worth noting they faced an issue with the parachain sov account not having funds in the AH triggering the account conversion issue (i recommend you to check this).

However from our side, we don't have capacity to troubleshoot specific issues with parachains as the workload for offering such as service is considerable.

[AurevoirXavier]

From our side, the Paseo AH is an exact copy of the Polkadot AH, so from the configuration perspective we are pretty confident that everything is correct.

Recently we upgrade the runtime for our mainnet.

Pretty interesting. Koi is also an exact copy of Darwinia. And everything works well between Darwinia and Polkadot AH.