scalecodec.exceptions.RemainingScaleBytesNotEmptyException

[wzli]

Just testing out a storage map retrieval call, encountered:

substrateinterface.base:DEBUG Connecting to ws://127.0.0.1:9944 ...
substrateinterface.base:DEBUG RPC request #1: "chain_getHead"
substrateinterface.base:DEBUG RPC request #2: "chain_getHeader"
substrateinterface.base:DEBUG RPC request #3: "chain_getRuntimeVersion"
substrateinterface.base:DEBUG RPC request #4: "state_getMetadata"
substrateinterface.base:DEBUG Retrieved metadata for 9 from Substrate node
substrateinterface.base:DEBUG RPC request #5: "state_getKeysPaged"
substrateinterface.base:DEBUG RPC request #6: "state_queryStorageAt"
Traceback (most recent call last):
  File "/home/wzli/projects/DecentralizedTaskAuction/bid_optimization/./substrate_test.py", line 33, in <module>
    contracts = substrate.query_map('Contracts', 'ContractInfoOf')
  File "/home/wzli/.local/lib/python3.9/site-packages/substrateinterface/base.py", line 1319, in query_map
    item_value = self.decode_scale(
  File "/home/wzli/.local/lib/python3.9/site-packages/substrateinterface/base.py", line 2656, in decode_scale
    obj.decode()
  File "/home/wzli/.local/lib/python3.9/site-packages/scalecodec/base.py", line 390, in decode
    raise RemainingScaleBytesNotEmptyException('Current offset: {} / length: {}'.format(self.data.offset, self.data.length))
scalecodec.exceptions.RemainingScaleBytesNotEmptyException: Current offset: 100 / length: 116

Versions of stuff:

   Installed package `canvas-node v0.1.0 (https://github.com/paritytech/canvas-node.git?tag=v0.1.5#385c4cc8)` (executable `canvas`)

Requirement already satisfied: substrate-interface==0.12.5 in /home/wzli/.local/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (0.12.5)
Requirement already satisfied: xxhash>=1.3.0 in /usr/local/lib64/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: requests~=2.25.1 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (2.25.1)
Requirement already satisfied: py-sr25519-bindings~=0.1.2 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (0.1.2)
Requirement already satisfied: certifi>=2020.11.8 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (2020.12.5)
Requirement already satisfied: scalecodec~=0.10.76 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (0.10.77)
Requirement already satisfied: idna>=2.8 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (2.8)
Requirement already satisfied: py-ed25519-bindings~=0.1.2 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (0.1.2)
Requirement already satisfied: websocket-client~=0.58.0 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (0.58.0)
Requirement already satisfied: py-bip39-bindings~=0.1.6 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (0.1.6)
Requirement already satisfied: base58~=2.0.1 in /home/wzli/.local/lib/python3.9/site-packages (from substrate-interface==0.12.5->-r requirements.txt (line 1)) (2.0.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/wzli/.local/lib/python3.9/site-packages (from requests~=2.25.1->substrate-interface==0.12.5->-r requirements.txt (line 1)) (1.25.3)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/lib/python3.9/site-packages (from requests~=2.25.1->substrate-interface==0.12.5->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: more-itertools in /home/wzli/.local/lib/python3.9/site-packages (from scalecodec~=0.10.76->substrate-interface==0.12.5->-r requirements.txt (line 1)) (8.7.0)
Requirement already satisfied: six in /usr/lib/python3.9/site-packages (from websocket-client~=0.58.0->substrate-interface==0.12.5->-r requirements.txt (line 1)) (1.15.0)

[arjanz]

Maybe the canvas type registry is outdated, I'll look into that soon

[LefterisJP]

Kusama balances for rotki also stopped working just now with a similar error. It has been working fine until this morning.

    def test_get_account_balance(kusama_manager):
>       balance = kusama_manager.get_account_balance(SUBSTRATE_ACC1_KSM_ADDR)

rotkehlchen/tests/unit/test_substrate_manager.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
rotkehlchen/chain/substrate/manager.py:88: in wrapper
    result = func(manager, *args_, **kwargs_)
rotkehlchen/chain/substrate/manager.py:543: in get_account_balance
    node_interface=node_interface,
rotkehlchen/chain/substrate/manager.py:290: in _get_account_balance
    params=[account],
../../.virtualenvs/rotkipy37/lib/python3.7/site-packages/substrateinterface/base.py:1237: in query
    obj.decode()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <accountinfo<index, accountdata>(value={'nonce': 796, 'consumers': 1, 'providers': 1, 'data': {'free': 343989901983444...2, 'reserved': 91753386293303529439232, 'miscFrozen': 85899345920000000000000, 'feeFrozen': 42949672960000000000000}})>, check_remaining = True

    def decode(self, check_remaining=True):
        self.data_start_offset = self.data.offset
        self.value = self.process()
        self.data_end_offset = self.data.offset

        if check_remaining and self.data.offset != self.data.length:
>           raise RemainingScaleBytesNotEmptyException('Current offset: {} / length: {}'.format(self.data.offset, self.data.length))
E           scalecodec.exceptions.RemainingScaleBytesNotEmptyException: Current offset: 76 / length: 80

../../.virtualenvs/rotkipy37/lib/python3.7/site-packages/scalecodec/base.py:383: RemainingScaleBytesNotEmptyException

This affects all rotki instances currently downloaded by our users. Unfortunately the developer who initially wrote the interface is not working with us anymore and as such I have to figure it out myself. What I don't understand is how can any changes in the library break things for us when we have the version pinned to substrate-interface==0.11.16.

We query account balances in the same way your readme says and it worked fine until this morning.

https://github.com/rotki/rotki/blob/23f41e34c6ab37764405aec0f5644927deb67aea/rotkehlchen/chain/substrate/manager.py#L286-L291

[arjanz]

@LefterisJP Although you pinned the library is pinned to a certain version is no guarantee unfortunately. Kusama had a runtime upgrade and it seems the AccountInfo struct has been modified. At the moment it is necessary to maintain a type registry manually to reflect the decomposition of the types used per runtime.

I agree this is somewhat cumbersome but atm there is no other way. There are plans to let the node expose this info of the runtime (just like the metadata), but this is still in development:

https://github.com/paritytech/substrate/discussions/8370

I think this will be the only way to keep this sustainable, but what I've heard this is not trivial.

I'll go figure out the changes in the new runtime now

[arjanz]

@LefterisJP Type update Kusama runtime v2030 https://github.com/polkascan/py-scale-codec/releases/tag/v0.10.78

[LefterisJP]

Hey @arjanz thanks for your response and for the information. So the types are somehow pulled dynamically I see. We need to find a way around it as it may cause problems like we just saw over the weekend with kusama queries completely breaking.

Will this be pulled automatically by py-substrate-interface or is some action needed?

[arjanz]

@LefterisJP To be honest I don't think currently there is a way to prevent it.. Right now the libraries like py-substrate-interface and PolkadotJS try their best to plan ahead and keep up with type definition changes, but when they happen due to runtime upgrades and we miss something, code could break until the type definition in the libraries reflect the runtime again. But hopefully this manually upkeeping won't be necessary with the mentioned scale-info project release.

For example this change fixed the issue we were facing today: https://github.com/polkascan/py-scale-codec/commit/e8f622341338132eca912bbb33b2194486085f4d

I think I can recall you guys use the use_remote_preset feature (see https://github.com/polkascan/py-substrate-interface#keeping-type-registry-presets-up-to-date) so as soon as we fix the type mismatches the new type registry will be downloaded and used.

Otherwise you need to bump the version of the py-scalecodec version to latest version (which should be done periodically anyway).

[LefterisJP]

Otherwise you need to bump the version of the py-scalecodec version to latest version (which should be done periodically anyway).

We don't have this pinned at all. We only have py-substrate-inerface pinned.

And yes we do use use_remote_preset=True. That's good then cause we can't bump any library version for a local app that has already been downloaded by the user.

Please let me know when the type mismatches are fixed so we can test it. Will the version we have work when you fix the types? substrate-interface==0.11.16?

@LefterisJP To be honest I don't think currently there is a way to prevent it.. Right now the libraries like py-substrate-interface and PolkadotJS try their best to plan ahead and keep up with type definition changes, but when they happen due to runtime upgrades and we miss something, code could break until the type definition in the libraries reflect the runtime again. But hopefully this manually upkeeping won't be necessary with the mentioned scale-info project release.

As long as it can be fixed by something not requiring a complete new re-release of our application then that is fine. Local apps like rotki can't afford to make releases too often as it's a process for users to upgrade.

[arjanz]

@LefterisJP I understand, that is exactly why we added the use_remote_preset feature after discussing your use-case with your developer sometime ago.

It should be fixed now, we committed and released an update (of py-scalecodec package)

[LefterisJP]

Hey @arjanz it does indeed seem fixed now. Thank you. Is there by the way another channel I can reach you from time to time with questions? I tried kusama and polkadot discord and also twitter but no luck.

[wzli]

@arjanz I found the issue regarding the opening post. It was because py-scalecodec was not updated to the type changes in pallets-contract v3.0, which is what the canvas node was using.

In particular, I was trying to query for ContractInfo type, which changed from: https://substrate.dev/rustdocs/v2.0.0/pallet_contracts/struct.RawAliveContractInfo.html to: https://substrate.dev/rustdocs/v3.0.0/pallet_contracts/struct.RawAliveContractInfo.html

So, I guess consider this a feature request? to support up-to-date contract-pallet and maybe some form of backwards compatibility?

[LefterisJP]

@wzli would using use_remote_preset=True help in your case? That should try to pull the types from the remote so you don't need to update manually each time a change happens.

[wzli]

I did try use_remote_preset=True it didn't work cause the upstream type_registry is outdated. the contract pallet interfaces seem to be defined in default.json so work if that was updated, but maybe there's also other stuff dependent on contract-pallet 2.0 idk. https://github.com/polkascan/py-scale-codec/blob/master/scalecodec/type_registry/default.json

[arjanz]

@wzli I have found a type decomposition difference in AliveContractInfo and AccountInfo types which will solve your specific issue, you will have to update your scalecodec package to https://github.com/polkascan/py-scale-codec/releases/tag/v0.10.80

However, I also saw the canvas nodes updated their calls and perhaps storage structure as well and the contract interface classes like ContractInstance and ContractCode also need to be updated for them to work with the new contract-pallet

[wzli]

Tried scalecodec 0.10.80, ran into the same issue.

What worked for was changing AliveContractInfo from

trie_id: TrieId
storage_size: u32
empty_pair_count: u32
total_pair_count: u32
code_hash: CodeHash
rent_allowance: Balance
deduct_block: BlockNumbe
last_write: Option<BlockNumber> 

to

trie_id: TrieId
storage_size: u32
pair_count: u32
code_hash: CodeHash
rent_allowance: Balance
rent_payed: Balance
deduct_block: BlockNumber
last_write: Option<BlockNumber>

I guess the different was pair count fields and added rent payed field.

You are also right about ContractInstance and ContractCode needing to be updated. When I tried contract_instance.read() the error is:

{'jsonrpc': '2.0', 'method': 'contracts_call', 'params': [{'dest': '0xc22ecb44ac929e5f60f60ab5e8ab0434d5ab97e962aea8a07a70f9dd28401c51', 'gasLimit': 5000000000000, 'inputData': '0x2f865bd9', 'origin': '5GC1pmXUB2XfS8XFMWYJMDWmXps4KqHwRnQ6hQaRppzxrnLP', 'value': 0}], 'id': 9}
Traceback (most recent call last):
  File "/home/wzli/projects/DecentralizedTaskAuction/bid_optimization/./substrate_test.py", line 42, in <module>
    result = auction.read(keypair, 'get')
  File "/home/wzli/.local/lib/python3.9/site-packages/substrateinterface/contracts.py", line 611, in read
    response = self.substrate.rpc_request(method='contracts_call', params=[{
  File "/home/wzli/.local/lib/python3.9/site-packages/substrateinterface/base.py", line 521, in rpc_request
    raise SubstrateRequestException(message['error'])
substrateinterface.exceptions.SubstrateRequestException: {'code': -32602, 'message': 'Invalid params: BadBase58.'}

I'm on the canvas version linked by the current ink docs https://paritytech.github.io/ink-docs/getting-started/setup

[arjanz]

@wzli To keep the issues organized I moved this issue to: https://github.com/polkascan/py-substrate-interface/issues/101

[wzli]

okay, thing work with the change to update ink versions. thank you.