Fetch aux store failed error - 'Not enough data to fill buffer' and 'out of range decoding Compact<u32>'

[NZT48]

Description

After upgrading Parachain dependencies (from Polkadot v0.9.27 to v0.9.40 -> PR), we were unable to access transaction receipt and other info for transactions that were executed before the upgrade, all EVM TXs that were executed after upgrade are accessible. We get to different errors Not enough data to fill buffer and out of range decoding Compact<u32> as response, and no error logs have been shown in parachain logs.

This applies to getTransactionByHash, getTransactionReceipt, getBlockByHash...

Steps to Reproduce

Replace the example steps below with actual steps to reproduce the bug you're reporting.

1. Go to OTP Polkadot.js
2. Click on eth -> getTransactionByHash(hash)
3. Provide this hash as input 0x71c2e10311e4181045ac872f3bb6f2e2a763b5423bd030cbae93fcea6e02ad48 (this TX was executed before the upgrade)
4. See error

Expected vs. Actual Behavior

It is expected to return transaction data as for example for this TX hash 0xdc3934d798d2f007a91c0263787844d5b438a9142d8c0b3dc9e98a3ead254a37 (this TX was executed after the upgrade).

Environment

• Operating system: Ubuntu
• Parachain github repo: https://github.com/OriginTrail/origintrail-parachain
• Frontier repo fork: https://github.com/OriginTrail/frontier/tree/polkadot-v0.9.40

Logs, Errors or Screenshots

Not enough data to fill buffer error log:

-32603: Call(Custom(ErrorObject { code: InternalError, message: "fetch aux store failed: \"Error { cause: None, desc: \\\"Not enough data to fill buffer\\\" }\"", data: None }))

Out of range error log:

-32603 : Call(Custom(ErrorObject { code: InternalError, message: "fetch aux store failed: \"Error { cause: None, desc: \\\"out of range decoding Compact<u32>\\\\" }\\"", data: None }))

Additional Information

No additional information at the moment, I am available to provide you with any additional info that you need.

[sorpaas]

This is clearly a decoding failure. What's your PALLET_ETHEREUM_SCHEMA on-chain?

[NZT48]

Thanks for responding @sorpaas. Honestly, I am not sure how to check that, I would be grateful if you would provide me step by step instructions.

I just found this, but I am not sure if it is useful, we are initializing overrides like this:

let overrides = fc_storage::overrides_handle(client.clone());

And this function overrides_handle from frontier inserts EthereumStorageSchema V1, V2 and V3.

[NZT48]

Hi @sorpaas, did I provide enough info about PALLET_ETHEREUM_SCHEMA?

[NZT48]

We are still unsure how to tackle this issue

[arthur-cw]

hey @NZT48 do you have any updates on this? I need to upgrade a Frontier-based solochain very much like you did here.

[dnjscksdn98]

In my case, due to the Frontier DB structure update, the Frontier DB migration had to be processed to version 1 to version 2. After the migration it now worked as expected.

[arthur-cw]

@dnjscksdn98 did you need to do anything besides running the forkless upgrade?

[dnjscksdn98]

@arthur-cw Instead on a forkless upgrade, the file mentioned below has to be removed and then restart your node client. {base-path}/chains/{chain-id}/frontier/db/db_version

On restart it will then print some logs such as this on the beginning.

✔️ Successful Frontier DB migration from version 1 to version 2

[arthur-cw]

@dnjscksdn98 did you do the same upgrade as reported by @NZT48 here?

I just upgraded my Frontier to polkadot-v0.9.43 and now I'm facing the same error for some transactions:

{
    "jsonrpc": "2.0",
    "error": {
        "code": -32603,
        "message": "Call(Custom(ErrorObject { code: InternalError, message: \"fetch aux store failed: \\\"Error { cause: None, desc: \\\\\\\"Not enough data to fill buffer\\\\\\\" }\\\"\", data: None }))"
    },
    "id": "0"
}

When I get the raw storage for key 0x3a657468657265756d5f736368656d61, which is :ethereum_schema I get 0x3, that is version 3. On the other hand, the db_version of frontier database is 2.

@dnjscksdn98 I will try your approach of deleting the db_version.

[arthur-cw]

@sorpaas can we get some feedback here? It would be good to at least understand better what's going on, specially for parachain teams.