"unknown block number" error from debug_traceCall RPC call against "latest" block during reorg

[gbrew]

Describe the bug

I make frequent debug_traceCall calls to reth, using the CallTracer, and specifying that they be made against the latest block. This usually works as expected, but I have noticed that I get an "unknown block number" RPC error sometimes when there is a reorg, instead of the call succeeding as expected.

I have observed a very similar issue when making eth_feeHistory calls (also specifying 'latest' block, 10 blocks previous, and explicitly specifying percentiles as [50]). These should clearly always succeed, but I get the "unknown block number" error sometimes during reorgs.

It appears that both these RPC methods end up seeing inconsistent state during a reorg. I suspect this is an issue in a number of other places as well. I'm guessing the issue is that 'latest' gets resolved to a specific block number or hash before the reorg, then that is no longer valid post-reorg. Please let me know if there's any other info which would be helpful tracking this down.

I've confirmed a reorg in the reth logs (added below) when I get the error.

Steps to reproduce

1. Sync reth
2. Make debug_traceCall RPC calls in a tight loop. I ran 1000 simultaneous tokio tasks calling in a loop.
3. Wait for a reorg, watch for RPC error

I've also tried reproducing this using eth_feeHistory calls as mentioned above. I've seen it happen, but it does seem to be much less frequent.

Here's the function I was running in a loop:

async fn call_rpc<M: Middleware>(provider: &M){
    let tx: TypedTransaction = TransactionRequest::new()
        .from(Address::zero())
        .to(Address::zero())
        .value(0u64).into();

    let tracing_options = GethDebugTracingOptions {
        tracer: Some(GethDebugTracerType::BuiltInTracer(GethDebugBuiltInTracerType::CallTracer)),
        tracer_config: Some(GethDebugTracerConfig::BuiltInTracer(GethDebugBuiltInTracerConfig::CallTracer(CallConfig{only_top_call: Some(false), with_log: Some(true)}))),
        ..Default::default()
    };
    let tracing_options = GethDebugTracingCallOptions { tracing_options };

    let _debug_trace = provider.debug_trace_call(tx, None, tracing_options).await.unwrap();
}

Node logs

2023-10-30T02:03:31.973880Z  INFO reth::cli: Status connected_peers=125 latest_block=18459877                                                                2023-10-30T02:03:41.212326Z  INFO reth::node::events: Block added to canonical chain number=18459878 hash=0xed223ec5304ea25973c96397b47d0ab3d28eaad8cb813ef07
71ca54a8177d95c                                                                                                                                              2023-10-30T02:03:41.749588Z  INFO reth::node::events: Canonical chain committed number=18459878 hash=0xed223ec5304ea25973c96397b47d0ab3d28eaad8cb813ef0771ca54a8177d95c elapsed=364.648187ms
2023-10-30T02:03:41.749700Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xed223ec5304ea25973c96397b47d0ab3d28eaad8cb813ef0771ca54a8177d95c s
afe_block_hash=0x17f43c42dc9a591799f4c995c28d5a9e56f2a81497b51949a7ee21c1ab4c297c finalized_block_hash=0x4fa453a4f9ec2774109053a9a73e4903ef17ab1673a0984505b20ee7054aa74d status=Valid
2023-10-30T02:03:50.627608Z  INFO reth::node::events: Block added to fork chain number=18459878 hash=0x5533cd2c52bde4f8cb6f8b13d1fad3bf3784f9edb40981848b60bf5fed111c55
2023-10-30T02:03:51.416693Z  INFO blockchain_tree: Unwinding canonical chain blocks: 18459878..=18459878                                                     2023-10-30T02:03:52.428641Z  INFO reth::node::events: Canonical chain committed number=18459878 hash=0x5533cd2c52bde4f8cb6f8b13d1fad3bf3784f9edb40981848b60bf5fed111c55 elapsed=1.011969647s
2023-10-30T02:03:52.428657Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x5533cd2c52bde4f8cb6f8b13d1fad3bf3784f9edb40981848b60bf5fed111c55 safe_block_hash=0x17f43c42dc9a591799f4c995c28d5a9e56f2a81497b51949a7ee21c1ab4c297c finalized_block_hash=0x4fa453a4f9ec2774109053a9a73e4903ef17ab1673a0984505b2
0ee7054aa74d status=Valid                                                                                                                                    2023-10-30T02:03:56.974077Z  INFO reth::cli: Status connected_peers=125 latest_block=18459878
2023-10-30T02:04:02.740728Z  INFO reth::node::events: Block added to canonical chain number=18459879 hash=0xf8dee0c138476a60af56a739e880236927652639fb1592c0f3a9a41e

### Platform(s)

Linux (x86)

### What version/commit are you on?

reth Version: 0.1.0-alpha.10 Commit SHA: a9fa281816e02c1ba936c65b290b4d20fcf83a6b

### What database version are you on?

Current database version: 1

### What type of node are you running?

Archive (default)

### What prune config do you use, if any?

none

### If you've built Reth from source, provide the full command you used

cargo install --locked --path bin/reth --bin reth

### Code of Conduct

- [X] I agree to follow the Code of Conduct

[mattsse]

I suspect this is an issue in a number of other places as well. I'm guessing the issue is that 'latest' gets resolved to a specific block number or hash before the reorg

I believe this is correct yes, need to check how we can mitigate this

[github-actions[bot]]

This issue is stale because it has been open for 21 days with no activity.

[toolchainx]

Same issue here, but with the eth_call method and block number set to latest.
docker image and reth version : ghcr.io/paradigmxyz/reth:v1.0.4

request body

{
  "method": "eth_call",
  "params": [
    {
      "to": "0x1116898dda4015ed8ddefb84b6e8bc24528af2d8",
      "data": "0x66c0bd240000000000000000000000006b175474e89094c44da98b954eedeac495271d0f"
    },
    "latest"
  ],
  "id": 213095,
  "jsonrpc": "2.0"
}

response body:

{
  "jsonrpc": "2.0",
  "id": 213095,
  "error": {
    "code": -32001,
    "message": "unknown block number"
  }
}