Help minimizing latency

[jajaislanina]

Hi erpc team,

I have a setup where i am running 2 Full nodes and 2 Archive nodes on Base Mainnet. Due to hardware limitations i need to use Full nodes to get the latest data because they are faster to sync and less prone to lag induced by high TPS on the network. My main use case is quick discovery of new blocks and re-executing all Tx from block. This causes ~4k-5k RPS to my node and is latency sensitive as i need to fetch all states from node before the next block is minded (~2 seconds on Base).

My goal is to make the erpc as fast as possible but to be able to handle errors when my application falls out of the range of the full node (128 blocks).

What i'm seeing is latency issues when using erpc compared to going directly to the node. While some delay is expected i am seeing 5x and more. On the left graph we see total avg request duration for eth_getStorageAt and on the right avg duration of the requests for same method from erpc to the node. Left graph definition:

/
sum(rate(erpc_network_request_duration_seconds_count{project='base'}[2m])) by (category)

Right graph definition:

sum(rate(erpc_upstream_request_duration_seconds_sum{project='base'}[2m])) by (category)
/
sum(rate(erpc_upstream_request_duration_seconds_count{project='base'}[2m])) by (category)

This is my erpc config

logLevel: info
database:
  evmJsonRpcCache:
    driver: memory
    maxItems: 10000

server:
  httpHost: 0.0.0.0
  httpPort: 4000

metrics:
  enabled: true
  host: 0.0.0.0
  port: 4001

projects:
  - id: base
    networks:
      - architecture: evm
        evm:
          chainId: 8453
        routing:
          scoreMultipliers:
          - blockHeadLag: 8
            errorRate: 1
            finalizationLag: 1
            method: '*'
            network: '*'
            overall: 1
            p90latency: 4
            throttledRate: 1
            totalRequests: 1
        failsafe:
          hedge:
            delay: 1000ms
            maxCount: 1
          retry:
            backoffFactor: 0.2
            backoffMaxDelay: 0s
            delay: 0ms
            jitter: 0ms
            maxCount: 2
          timeout:
            duration: 3s

    upstreams:
      - id: base-full-0
        endpoint: http://node-base-mainnet-full-0:8545
        evm:
           chainId: 8453
           nodeType: full
      - id: base-full-1
        endpoint: http://node-base-mainnet-full-1:8545
        evm:
           chainId: 8453
           nodeType: full
      - id: base-base-0:8545
        endpoint: http://node-base-mainnet-0:8545
        evm:
          chainId: 8453
          nodeType: archive
      - id: base-archive-1
        endpoint: http://node-base-mainnet-1:8545
        evm:
          chainId: 8453
          nodeType: archive

Any tips on how to tune the config to optimize low latency?

[aramalipoor]

Hey @jajaislanina

There a few configs you can play around to reduce overhead to minimum:

• Use directiveDefaults under project.networkDefaults.directiveDefaults to disable "retry" for pending txs or empty results
• Disabling cache means no JSON traversing and can help with lower latency as well just set https://docs.erpc.cloud/faq#how-to-disable-caching database.evmJsonRpcCache: ~
• You can also playaround with failsafe.hedge as well, basically firing off more concurrent requests so that fastest node response is used.

I'm happy to pair with you to investigate further and get it to lowest overhead possible (normally i don't expect more than 5ms overhead to be necessary when things are properly initialized) https://t.me/erpc_cloud DM me :)