search

ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.39k stars 20.05k forks source link

Timed out on `eth_getLogs` request through large range of blocks #28765

Open nordschesasa opened 9 months ago

nordschesasa commented 9 months ago

Hi, I've been using Geth as my ETH fullnode, it may get timed out on eth_getLogs request through large range of blocks, for example with request:

{
  "method": "eth_getLogs",
  "params": [
    {
      "fromBlock": "0x0",
      "toBlock": "0x120dc53",
      "address": "0xb62bcd40a24985f560b5a9745d478791d8f1945c",
      "topics": [
        [
          "0xcfb473e6c03f9a29ddaf990e736fa3de5188a0bd85d684f5b6e164ebfbfff5d2"
        ]
      ]
    }
  ],
  "id": 62,
  "jsonrpc": "2.0"
}

Response from Node:

{
    "jsonrpc": "2.0",
    "id": 62,
    "error": {
        "code": -32002,
        "message": "request timed out"
    }
}

However if the block range is small, it can response quickly without a problem, I've tried Infura/alchemy with same request and it can response quickly with data:

{
    "jsonrpc": "2.0",
    "id": 62,
    "result": [
        {
            "address": "0xb62bcd40a24985f560b5a9745d478791d8f1945c",
            "blockHash": "0xb1c05e3a5f7791b40d9ded2bb67bd2d250f1ccb036dac0f6a046b7ed2d416df0",
            "blockNumber": "0xed6e42",
            "data": "0x0000000000000000000000006b7763b749073e892c83e674c1ec4799d6f339ef",
            "logIndex": "0x151",
            "removed": false,
            "topics": [
                "0xcfb473e6c03f9a29ddaf990e736fa3de5188a0bd85d684f5b6e164ebfbfff5d2"
            ],
            "transactionHash": "0x87130dfe52f1eb4ec22261333534ee7ac2c15e5256ffa7a59ae7153119c6cd73",
            "transactionIndex": "0xc9"
        },
        {
            "address": "0xb62bcd40a24985f560b5a9745d478791d8f1945c",
            "blockHash": "0xb1c05e3a5f7791b40d9ded2bb67bd2d250f1ccb036dac0f6a046b7ed2d416df0",
            "blockNumber": "0xed6e42",
            "data": "0x0000000000000000000000006ea99c6fe2c770c2c46ebe03a4855977282e844f",
            "logIndex": "0x153",
            "removed": false,
            "topics": [
                "0xcfb473e6c03f9a29ddaf990e736fa3de5188a0bd85d684f5b6e164ebfbfff5d2"
            ],
            "transactionHash": "0xca00caf8ad277bd23597d497e1c77b4308d40e8787d7e0e5204d320f1f3ab31c",
            "transactionIndex": "0xcb"
        }
    ]
}

By using web3_clientVersion to query their(infura and alchemy) node's version I can see they are using Geth as well(version 1.13.4 and 1.13.5).

I've seen there is a post on Infura, not sure if related to this, https://www.infura.io/blog/post/faster-logs-and-events-e43e2fa13773, I might wonder if this kind of request cannot be handled by Geth itself.

The node it's running with docker compose, with file as below:

version: "3"
services:
  geth:
    image: ethereum/client-go:v1.13.5
    restart: unless-stopped
    ports:
      - 30303:30303
      - 30303:30303/udp
      - 127.0.0.1:8545:8545
      - 127.0.0.1:8546:8546
      - 127.0.0.1:8551:8551
    volumes:
      - ./data:/root/.ethereum
    command:
      - --http
      - --cache=8192
      - --http.api=eth,net,web3,engine,admin
      - --http.addr=0.0.0.0
      - --http.vhosts=*
      - --http.corsdomain=*
      - --maxpeers=200
      - --ws
      - --ws.origins=*
      - --ws.addr=0.0.0.0
      - --ws.api=eth,net,web3
      - --graphql
      - --graphql.corsdomain=*
      - --graphql.vhosts=*
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/root/.ethereum/jwt.hex
      - --authrpc.vhosts=*
      - --authrpc.port=8551
      - --history.transactions=0

  prysm:
    image: gcr.io/prysmaticlabs/prysm/beacon-chain:latest
    pull_policy: always
    container_name: beacon
    restart: unless-stopped
    stop_grace_period: 2m
    volumes:
      - ./prysm_data:/data
      - ./data:/geth
    depends_on:
      geth:
        condition: service_healthy
    ports:
      - 127.0.0.1:4000:4000
      - 127.0.0.1:3500:3500
    command:
      - --accept-terms-of-use
      - --datadir=/data
      - --disable-monitoring
      - --rpc-host=0.0.0.0
      - --execution-endpoint=http://geth:8551
      - --jwt-secret=/geth/jwt.hex
      - --rpc-host=0.0.0.0
      - --rpc-port=4000
      - --grpc-gateway-corsdomain=*
      - --grpc-gateway-host=0.0.0.0
      - --grpc-gateway-port=3500
      - --checkpoint-sync-url=https://mainnet-checkpoint-sync.attestant.io/
      - --genesis-beacon-api-url=https://mainnet-checkpoint-sync.attestant.io/

The server is hosted on GCP with 4T of SSD persistent disk so disk IO shouldn't be a problem here.

s1na commented 9 months ago

This range:

      "fromBlock": "0x0",
      "toBlock": "0x120dc53",

is basically the whole history of ethereum. I understand this is probably a topic that doesn't occur often. Geth right now doesn't optimize for the "sparse logs" scenario as much.

There are some ideas on the backburner for optimizing getLogs https://github.com/ethereum/go-ethereum/issues/25336 but not much progress.

Something coming in the near future tho is the ability to subscribe to historical logs https://github.com/ethereum/go-ethereum/pull/27439 so at least for such queries that take a while it doesn't time-out. It will be a long-running process within geth that will deliver logs via websocket as it's searching the history.