Getting "tracing failed: nonce too high" when running debug_traceBlockByHash at the tip

[quick-pawiromitchel]

System information

Erigon version: erigon/2.54.0/linux-amd64/go1.20.10

OS & Version: Linux

Consensus Layer: Lighthouse v4.3.0-dfcb336

Chain/Network: ETH Mainnet

Expected behaviour

To give full debug trace results

Actual behaviour

Getting these errors inside the response

{
  error: {
    code: -32000,
    message: 'tracing failed: nonce too high: address 0x0000000000000000000000000000000000000000, tx: 137 state: 0'
  }
}

Happened at these blocks:

• 0x7de57820bb4e00876d0269695fcf2c47d28efcde4f43c4062dff96d80f6f6a78
• 0x787d9cdd803ab6a37ea7d888a8a9dad633127f94ddd321cb935f508673a3d846

Steps to reproduce the behaviour

• subscribe to incoming blocks
• run debug_traceBlockByHash at the tip

Script

const Web3 = require('web3');
const fetch = require('node-fetch');

async function startConnection() {
    const options = {
        timeout: 30000, // ms

        clientConfig: {
            // Useful if requests are large
            maxReceivedFrameSize: 100000000,   // bytes - default: 1MiB
            maxReceivedMessageSize: 100000000, // bytes - default: 8MiB

            // Useful to keep a connection alive
            keepalive: true,
            keepaliveInterval: 60000 // ms
        },

        // Enable auto reconnection
        reconnect: {
            auto: true,
            delay: 5000, // ms
            maxAttempts: 5,
            onTimeout: false
        }
    };

    const web3 = new Web3("ws://", options);
    let ws = web3.eth.subscribe('newBlockHeaders')

    ws.on("connected", function (subscriptionId) {
        console.log('connected to WSS ID:', subscriptionId);
    })

    ws.on("data", async function (block) {
        const blockHash = block.hash;

        console.log(`incoming ${blockHash}`);

        const data = {
            method: "debug_traceBlockByHash",
            params: [blockHash, { "tracer": "callTracer", "timeout": "240s" }],
            jsonrpc: "2.0",
            id: new Date().getTime()
        };

        fetch(url, {
            method: 'POST',
            body: JSON.stringify(data),
            headers: { 'Content-Type': 'application/json' }
        })
            .then(response => response.json())
            .then(result => {
                result.result.forEach(element => {
                    if (element.result === null) {
                        console.log(url, element.result);
                    }

                    // if the error field is present
                    if (element.error) {
                        console.log(url, element)
                    }
                });

                if (result.error) {
                    console.log(url, result.error);
                }
            })
            .catch(error => {
                console.log(url, error);
            });
    });
}

startConnection();

[quickchase]

bump

[AskAlexSharov]

I didn't reproduce yet. But I have some guess: Erigon does store state history only for Canonical blocks. It means you can observe race:

• you receive newHead with hash X1
• erigon does re-org and X1 is not-canonical anymore
• you send trace_block(X1) - erigon fail because doesn't have state history for this block
• erigon re-org back to fork where X1 canonical
• you send trace_block(X1) - erigon succeed

So, I will add isCanonicalHash check before start tracing.

[AskAlexSharov]

Yes, seems debug_traceBlockByHash didn't check isCanonical. Adding: https://github.com/ledgerwatch/erigon/pull/9616

[AskAlexSharov]

fixed one related problem: https://github.com/ledgerwatch/erigon/pull/9684

[alfredolopez80]

this problem stil present i have a services with Quick Node and i have the issue random,

curl https://damp-flashy-waterfall.xdai.quiknode.pro/f6679a16b04a01a745a92052eaa19f71f686b7cb/ \ 17:23:58 -X POST \ -H "Content-Type: application/json" \ --data '{"method":"trace_block","params":["0x1f4aef0"],"id":1,"jsonrpc":"2.0"}'

when i request twice fail!!!

Another Exam: ValueError: Problem with payload={'id': 6, 'jsonrpc': '2.0', 'method': 'trace_block', 'params': ['0x1f42532']}result={'jsonrpc': '2.0', 'id': 6, 'error': {'code': -32000, 'message': 'first run for txIndex 0 error: nonce too high: address 0x0000000000000000000000000000000000000000, tx: 26891 state: 0'}}

when request about block 32777522

and this repeat with blocks:

• 32777411
• 32777446
• 32777452
• 32777484
• 32777498

and is the same behavior

[AskAlexSharov]

@alfredolopez80 hi, we released couple related fixes in v2.59 please try. thank you.

[DmytroShalaiev]

bump

[AskAlexSharov]

@DmytroShalaiev we released couple related fixes in v2.59 please try. thank you.

[yperbasis]

Please re-open if it still happens with v2.60.2

[DmytroShalaiev]

Still relevant on (Quicknode, Blast, Chainstack)

20719908 block in ethereum

"code":-32000,"message":"tracing failed: nonce too high: address 0x0000000000000000000000000000000000000000, tx: 13 state: 0"