silent fail or corrupted state from XCM calls?

[spazcoin]

I'm having issues constructing XCM calls. I've read the XCM sections of the Polkadot wiki, Moonbeam XCM guide, and README in the xcm-format repo but I still find it difficult to debug the failures, so I'm hoping you guys know better ways.

I've set LOG_LEVEL="debug" and I start Chopsticks locally using the following command and then connect using Polkadot.js. npx @acala-network/chopsticks@latest xcm -r polkadot -p interlay -p centrifuge -p hydradx

I run each extrinsic string below from the JS console like:

const number = (await api.rpc.chain.getHeader()).number.toNumber()
await api.rpc('dev_setStorage', {
 scheduler: {
   agenda: [
     [
       [number + 1], [
         {
           call: {
             Inline: '0x4d047369626cd6070000000000000000000000000000000000000000000000000000090000001f00001062fde3f00184770400'
           },
           origin: {
             system: 'Root'
           }
         }
       ]
     ]
   ]
 }
})
await api.rpc('dev_newBlock', { count: 1})

[spazcoin]

Scenario 1: Transactions submitted on the Centrifuge chain. 1) xtransfer.send tokens from CFG treasury to CFG sibling acct on HydraDX: (successful) 0x3e0300016d6f646c70792f747273727900000000000000000000000000000000000000007c00000000904cbb5f69aad29e00000000000003010200c91f01007369626cef07000000000000000000000000000000000000000000000000000000 2) Call omnipool.addLiquidity(13, 46875 CFG) on Hydra using an XCM call from Centrifuge. I'm withdrawing CFG token from its sibling acct on Hydra to pay for the transaction, but I can't get the number of decimal places right. CFG is 18 decimals. 0x790003010100c91f0314000400010200bd1f060200010000000000000000000000000000000000000000000000000000000000000013000064a7b3b6e00d1300010200bd1f060200010000000000000000000000000000000000000000000000000000000000000013000064a7b3b6e00d00060013000064a7b3b6e00d6566583b020d00000000008c239bb14d19ed09000000000000140d0100010200c91f01007369626cef070000000000000000000000000000000000000000000000000000 When I withdraw/BuyExecution with 10^18 CFG then I get a Corrupted State error from Chopsticks.

[00:01:06.528] DEBUG (ws): New connection
    app: "chopsticks"
[00:01:06.602] DEBUG: dev_setStorage
    app: "chopsticks"
    hash: "0x85b6559328e15fb657163309823002b367c64bd46af44c6ceb56e766189cd396"
    values: {
      "scheduler": {
        "agenda": [
          [
            [
              5458439
            ],
            [
              {
                "call": {
                  "Inline": "0x790003010100c91f0314000400010200bd1f060200010000000000000000000000000000000000000000000000000000000000000013000064a7b3b6e00d1300010200bd1f060200010000000000000000000000000000000000000000000000000000000000000013000064a7b3b6e00d00060013000064a7b3b6e00d6566583b020d00000000008c239bb14d19ed09000000000000140d0100010200c91f01007369626cef070000000000000000000000000000000000000000000000000000"
                },
                "origin": {
                  "system": "Root"
                }
              }
            ]
          ]
        ]
      }
    }
runtime::storage             ERROR: Corrupted state at `[61, 183, 162, 76, 253, 201, 222, 120, 89, 116, 116, 108, 20, 169, 157, 249, 22, 67, 245, 65, 151, 24, 33, 156, 149, 103, 157, 221, 45, 130, 85, 116, 209, 208, 242, 103, 176, 28, 18, 128, 7, 74, 83, 0]: Error`
runtime::system              DEBUG: [5458439] 0 extrinsics, length: 3909 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 5546994779, proof_size: 53095) (0%)
[00:01:09.251] INFO (block-builder): Centrifuge building #5,458,439
    app: "chopsticks"
    number: 5458439
    extrinsics: []
    umpCount: 0
[00:01:15.996] INFO (block-builder): Centrifuge new head #5,458,439
    app: "chopsticks"
    number: 5458439
    hash: "0x05a2c8a424db9dadece517ef5d03ac7b8e6b7da5d8781d69c2fa06d54bd6cfc0"
    extrinsics: []
    pendingExtrinsics: []
    ump: {}
[00:01:15.997] DEBUG (blockchain): setHead
    app: "chopsticks"
    number: 5458439
    hash: "0x05a2c8a424db9dadece517ef5d03ac7b8e6b7da5d8781d69c2fa06d54bd6cfc0"
[00:01:15.997] INFO (xcm): outboundHrmpMessage
    app: "chopsticks"
    outboundHrmpMessage: []
[00:01:16.000] DEBUG: dev_newBlock
    app: "chopsticks"
    hash: "0x05a2c8a424db9dadece517ef5d03ac7b8e6b7da5d8781d69c2fa06d54bd6cfc0"

[spazcoin]

Scenario 2: Similar to Scenario 1, but calls from Interlay (INTR is 10 decimals) to see if CFG 18 decimals is causing issues. 1) xtransfer.send tokens from Interlay treasury to Interlay sibling acct on HydraDX: (successful) 0x030300016d6f646c6d6f642f7472737900000000000000000000000000000000000000005e000002008053ee7ba80a00000000000000000003010200c91f01007369626cf007000000000000000000000000000000000000000000000000000000 2) Call omnipool.addLiquidity(17, 2900 INTR) on Hydra using an XCM call from Interlay 0x5b0003010100c91f0310000400010200c11f06020002000000000000000000000000000000000000000000000000000000000000000700e40b54021300010200c11f06020002000000000000000000000000000000000000000000000000000000000000000700e40b54020006000700e40b5402feff0300583b02110000000040d9dd884d0a00000000000000000014 Note that withdrawing 10^10 is 1 INTR for fees. Result: no warnings/errors in Chopsticks. Interlay generates a new block but I don't see my XCM call in it.

[spazcoin]

Scenario 3: Transactions submitted on the HydraDX chain. Using Hydra governance to deposit funds into the Omnipool on behalf of a parachain's sibling acct. I can individually perform ASTR and BNC deposits using batchAll calls, but if I combine the two into a single call then it silently fails. No error on Chopsticks console, and no failure shown in Hydra block explorer. success: 0x0d020c4101090000000110270000102700000d0301017369626cd60700000000000000000000000000000000000000000000000000003b0209000000000000ffcf8529967e7704000000000041010900000001f401000010270000 success: silent fail: batch(batchAll(ASTR), batchAll(BNC)) 0x0d00080d020c4101090000000110270000e80300000d0301017369626cd60700000000000000000000000000000000000000000000000000003b0209000000000080be058e7400249504000000000041010900000001f4010000102700000d020c41010e0000000110270000e80300000d0301017369626cee0700000000000000000000000000000000000000000000000000003b020e00000000c057f42fd25811000000000000000041010e00000001f401000010270000 silent fail: batch(ASTR, BNC) 0x0d02184101090000000110270000e803000041010e0000000110270000e80300000d0301017369626cd60700000000000000000000000000000000000000000000000000003b0209000000000080be058e740024950400000000000d0301017369626cee0700000000000000000000000000000000000000000000000000003b020e00000000c057f42fd25811000000000000000041010900000001f40100001027000041010e00000001f401000010270000 result: (no error messages. new block doesn't contain the extrinsic)

[23:46:30.136] DEBUG (ws): New connection
    app: "chopsticks"
[23:46:30.211] DEBUG: dev_setStorage
    app: "chopsticks"
    hash: "0xcecca7601d77c0ea468eb1ef7b708590271d83165f63298395a8bc498565d2bd"
    values: {
      "scheduler": {
        "agenda": [
          [
            [
              5090707
            ],
            [
              {
                "call": {
                  "Inline": "0x0d00080d020c4101090000000110270000e80300000d0301017369626cd60700000000000000000000000000000000000000000000000000003b0209000000000080be058e7400249504000000000041010900000001f4010000102700000d020c41010e0000000110270000e80300000d0301017369626cee0700000000000000000000000000000000000000000000000000003b020e00000000c057f42fd25811000000000000000041010e00000001f401000010270000"
                },
                "origin": {
                  "system": "Root"
                }
              }
            ]
          ]
        ]
      }
    }
[23:46:30.534] INFO (block-builder): HydraDX building #5,090,707
    app: "chopsticks"
    number: 5090707
    extrinsics: []
    umpCount: 0
[23:46:45.513] INFO (block-builder): HydraDX new head #5,090,707
    app: "chopsticks"
    number: 5090707
    hash: "0xddd325984fe912a2bb158e760fca6dc615ee2e669da0825462498d849f7250b9"
    extrinsics: []
    pendingExtrinsics: []
    ump: {}
[23:46:45.513] DEBUG (blockchain): setHead
    app: "chopsticks"
    number: 5090707
    hash: "0xddd325984fe912a2bb158e760fca6dc615ee2e669da0825462498d849f7250b9"
[23:46:45.514] INFO (xcm): outboundHrmpMessage
    app: "chopsticks"
    outboundHrmpMessage: []
[23:46:45.528] DEBUG: dev_newBlock
    app: "chopsticks"
    hash: "0xddd325984fe912a2bb158e760fca6dc615ee2e669da0825462498d849f7250b9"
[23:46:49.904] DEBUG (ws): Connection closed
    app: "chopsticks"
[23:46:50.904] DEBUG (ws): New connection
    app: "chopsticks"

Question: shouldn't there be some sort of error/warning printed out by Chopsticks? Or is the issue with error handling somewhere in substrate? How can I debug these issues?

[qiweiii]

@spazcoin I just tried the Scenario 1 and 2, seems both cases have the same error. For interlay, it did not show the error because it does not has runtime-log-level: 5, which centrifuge has.

Maybe you could debug with trace log level, using local config files with runtime-log-level=5 (runtime log), and also LOG_LEVEL="trace" (node log)

[xlc]

Chopsticks only run the runtime. There isn't much we can do if the runtime had issue process the inputs. You will need to as the parachain team to fix whatever issue you having.

However there is a non zero chance that Chopsticks didn't 100% correctly executing the runtime so I can take a look this later.

[spazcoin]

Thank you both for your comments. runtime-log-level: 5 did end up being very useful in debugging the other chains, so I submitted #752 to make that the default setting for many other chains. I also tried LOG_LEVEL="trace" but it was too noisy for my taste.

Through more testing, I've determined that DOT in a parachain's sibling acct can be used to pay XCM transaction fees (and I've heard other sufficient tokens on AssetHub) but perhaps native parachain tokens cannot. (ongoing discussions on CFG forum and HydraDx Discord)

[spazcoin]

Update: jgreen drafted a Centrifuge call that successfully makes an Omnipool deposit and uses CFG tokens to pay for the XCM transaction fees. (scenario 1, step 2 above) 0x790003010100c91f0314000400010200bd1f060200010000000000000000000000000000000000000000000000000000000000000013000064a7b3b6e00d1300010200bd1f060200010000000000000000000000000000000000000000000000000000000000000013000064a7b3b6e00d0006010700e40b540202000400583b020d0000000000e8890423c78a0000000000000000140d010000010100d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d

However, when I run the same extrinsic on my locally running Chopsticks instance, I still get the Corrupted State error. So I started clean and re-downloaded Chopsticks, install dependencies and build. Run Chopsticks and try Scenario 1 with his call again and I still get the same error.

[ermalkaleci]

@spazcoin can you write a new instruction how to reproduce this. I tried to run this but getting badOrigin using Alice. Can it be in one page please?

[ermalkaleci]

ok I was able to reproduce

[ermalkaleci]

The corrupt state seems to be your scheduled call

[spazcoin]

@ermalkaleci thank you so much for investigating!!! I also was getting "Corrupted State" when running William Freude's CFG call so definitely an issue with just my setup.

So what can I do? What is a different code snippet that I should use in the p.js JS console to test executing draft extrinsics from root (for governance)?

[ermalkaleci]

@spazcoin inline call is bounded to 128 https://github.com/paritytech/polkadot-sdk/blob/d05786ffb5523c334f10d16870c2e73674661a52/substrate/frame/support/src/traits/preimages.rs#L29 your calls exceeds that

[ermalkaleci]

I suggest you write a script and use this as reference to schedule calls https://github.com/AcalaNetwork/chopsticks/blob/15ef2280951b1c93baa9239f4b32721b2085afff/packages/chopsticks/src/plugins/dry-run/dry-run-preimage.ts#L25

[spazcoin]

@spazcoin inline call is bounded to 128 https://github.com/paritytech/polkadot-sdk/blob/d05786ffb5523c334f10d16870c2e73674661a52/substrate/frame/support/src/traits/preimages.rs#L29 your calls exceeds that

@ermalkaleci this was the golden answer that I've been searching for for over a month. THANK YOU! I will definitely work on building TS scripts in the future as you suggested. However, I remembered that jgreen gave me a code snippet to execute a preimage. So in Polkadot.js/apps UI connected to Chopsticks I was able to submit my very long extrinsic as a preimage and then execute that preimage from root using:

const number = (await api.rpc.chain.getHeader()).number.toNumber()
await api.rpc('dev_setStorage', {
 scheduler: {
   agenda: [
     [
       [number + 1], [
         {
           call: {
             Lookup: {
               hash: '0x515b70543e1530cf269523853d5e3abb4c7220caa283ccd2c947fef7ef846ec9',
               len: 304
             }
           },
           origin: {
             system: 'Root'
           }
         }
       ]
     ]
   ]
 }
})
await api.rpc('dev_newBlock', { count: 22})

(count 22 because the one I'm testing uses Centrifuge's scheduler to loop 20 times through depositing CFG tokens into the HydraDX Omnipool)

Therefore I'm closing this issue. Thanks again ❤️