Handling 429 responses from rpc endpoint when using forking

[kowalski]

Version of Hardhat

2.22.3

What happened?

I can see that in new version of hardhat the provider using by hardhat network has been re-written to Rust. Nice job guys! Please correct me if I'm wrong, but my understanding is that currently the code making requests to the forking rpc endpoint resides here: https://github.com/NomicFoundation/hardhat/blob/main/crates/edr_eth/src/remote/client.rs#L484

I can see that this place is not handling the 429 response. As a result I'm getting a following error when running debug_traceTransaction against free Alchemy account:

{"errorType":"Error","errorMessage":"The response reported error `429`: `Your app has exceeded its compute units per second capacity. If you have retries enabled, you can safely ignore this message. If not, check out https://docs.alchemy.com/reference/throughput`. (optional data: None). Request: [{\"id\":40,\"jsonrpc\":\"2.0\",\"method\":\"eth_getBalance\",\"params\":[\"0x71be63f3384f5fb98995898a86b02fb2426c5788\",\"0x12c553f\"]},{\"id\":41,\"jsonrpc\":\"2.0\",\"method\":\"eth_getTransactionCount\",\"params\":[\"0x71be63f3384f5fb98995898a86b02fb2426c5788\",\"0x12c553f\"]},{\"id\":42,\"jsonrpc\":\"2.0\",\"method\":\"eth_getCode\",\"params\":[\"0x71be63f3384f5fb98995898a86b02fb2426c5788\",\"0x12c553f\"]}]","code":"GenericFailure","stack":["Error: The response reported error `429`: `Your app has exceeded its compute units per second capacity. If you have retries enabled, you can safely ignore this message. If not, check out https://docs.alchemy.com/reference/throughput`. (optional data: None). Request: [{\"id\":40,\"jsonrpc\":\"2.0\",\"method\":\"eth_getBalance\",\"params\":[\"0x71be63f3384f5fb98995898a86b02fb2426c5788\",\"0x12c553f\"]},{\"id\":41,\"jsonrpc\":\"2.0\",\"method\":\"eth_getTransactionCount\",\"params\":[\"0x71be63f3384f5fb98995898a86b02fb2426c5788\",\"0x12c553f\"]},{\"id\":42,\"jsonrpc\":\"2.0\",\"method\":\"eth_getCode\",\"params\":[\"0x71be63f3384f5fb98995898a86b02fb2426c5788\",\"0x12c553f\"]}]"]}

We're still using an older version of hardhat that had this logic written in packages/hardhat-core/src/internal/core/providers/http.ts module. In there exist is a logic that detects 429 responses, waits and retries. Is it within your plan to add an analogues logic to the Rust implementation?

Minimal reproduction steps

I'm getting this error when calling

await hardhatProvider.send('debug_traceTransaction', [txHash])

for

const txHash = "0x77aeaefa1fcf273ae1a20fcf5f86627986e677cf9bb13abbf17b6844e45dee64"

I'm using the following network config:

export const config: HardhatUserConfig = {
  solidity: '0.8.9',
  paths: {
    cache: '/tmp/cache',
  },
  networks: {
    hardhat: {
      chainId: 1,
      forking: {
        url: `https://eth-mainnet.alchemyapi.io/v2/${process.env.ALCHEMY_KEY}`,
      },
      chains: {
        // chains are configured to fork from blocks around the time of
        // March 15th 2023. it seems like a safe timespan for our use case.

        // CAUTION:
        // we shouldn't fork from too old blocks, because of getting errors of
        // too many requests per second
        [ChainId.polygon]: {
          hardforkHistory: {
            cancun: 0,
          },
        },
        [ChainId.mumbai]: {
          hardforkHistory: {
            cancun: 33_130_000,
          },
        },
        [ChainId.sepolia]: {
          hardforkHistory: {
            cancun: 3_100_000,
          },
        },
        [ChainId.arbitrum]: {
          hardforkHistory: {
            cancun: 70_000_000,
          },
        },
        [ChainId.arbitrumGoerli]: {
          hardforkHistory: {
            cancun: 11_500_000,
          },
        },
      },
    },
  },
}

Search terms

429 forking rate limit

[kowalski]

Ok, I need to update this issue a bit as I've been debugging this for few hours and I understand it's a bit more complex.

First, I retract a statement that there is no retry on 429 responses in new rust client. I can see now this bit:

impl RpcClient {
    /// Create a new instance, given a remote node URL.
    /// The cache directory is the global EDR cache directory configured by the
    /// user.
    pub fn new(
        url: &str,
        cache_dir: PathBuf,
        extra_headers: Option<HeaderMap>,
    ) -> Result<Self, RpcClientError> {
        let retry_policy = ExponentialBackoff::builder()
            .retry_bounds(MIN_RETRY_INTERVAL, MAX_RETRY_INTERVAL)
            .base(EXPONENT_BASE)
            .build_with_max_retries(MAX_RETRIES);

        ....
        #[cfg(not(feature = "tracing"))]
        let client = HttpClientBuilder::new(client)
            .with(RetryTransientMiddleware::new_with_policy(retry_policy))
            .build();

So it's done and it's done elegantly.

The issue is somewhere else. I've been going through the alchemy dashboard and I can see the following:

To make it more readable let me copy paste request:

[
  {
    "id": 52,
    "jsonrpc": "2.0",
    "method": "eth_getBalance",
    "params": [
      "0x8626f6940e2eb28930efb4cef49b2d1f2c9c1199",
      "0x12c59e0"
    ]
  },
  {
    "id": 53,
    "jsonrpc": "2.0",
    "method": "eth_getTransactionCount",
    "params": [
      "0x8626f6940e2eb28930efb4cef49b2d1f2c9c1199",
      "0x12c59e0"
    ]
  },
  {
    "id": 54,
    "jsonrpc": "2.0",
    "method": "eth_getCode",
    "params": [
      "0x8626f6940e2eb28930efb4cef49b2d1f2c9c1199",
      "0x12c59e0"
    ]
  }
]

and response:

[
  {
    "jsonrpc": "2.0",
    "id": 52,
    "error": {
      "code": 429,
      "message": "Your app has exceeded its compute units per second capacity. If you have retries enabled, you can safely ignore this message. If not, check out https://docs.alchemy.com/reference/throughput"
    }
  },
  {
    "jsonrpc": "2.0",
    "id": 53,
    "error": {
      "code": 429,
      "message": "Your app has exceeded its compute units per second capacity. If you have retries enabled, you can safely ignore this message. If not, check out https://docs.alchemy.com/reference/throughput"
    }
  },
  {}
]

And status code is 200, instead of 429.

Shame on you Alchemy!

[fvictorio]

Hi @kowalski, I think this is a duplicate of https://github.com/NomicFoundation/edr/issues/381, but thanks for sharing that info, it's good to have confirmation that we need to do this!