Avalanche - GetBlockHash error height 1: not found

[mattmartinez]

Description

I've been having issues trying to get my C chain to have block data from the beginning (genesis) so that blockbook can start syncing, but despite updating the chain-config-content to use:

{
  "snowman-api-enabled": false,
  "coreth-admin-api-enabled": false,
  "local-txs-enabled": true,
  "pruning-enabled": false,
  "eth-apis": [
    "internal-eth",
    "internal-blockchain",
    "internal-transaction",
    "internal-tx-pool",
    "internal-account",
    "internal-personal",
    "debug-tracer",
    "web3",
    "eth",
    "eth-filter",
    "admin",
    "net"
  ]
}

It continues to not build a full archive node, as I can't query the first genesis block. Is there something I'm missing in regards to the bootstrap process to ensure C chain archives everything since block 1?

Testing

using my python script i can confirm they are bootstrapped and responding with the latest block, so I know the chains are operational:

root@ip-10-0-0-84:~# cat ~/avax.py
import requests
import json

url = "http://127.0.0.1:8098/"
headers = {'content-type':'application/json;'}

def send_request(method, params, endpoint):
    data = {
        "jsonrpc": "2.0",
        "id": 1,
        "method": method,
        "params": params
    }
    response = requests.post(url + endpoint, data=json.dumps(data), headers=headers)
    return response.json()

# Get height for avm
avm_height = send_request("avm.getHeight", {}, "ext/bc/X")
print(f"AVM Height (X): {avm_height['result']['height']}")

# Get height for platform
platform_height = send_request("platform.getHeight", {}, "ext/P")
print(f"Platform Height (P): {platform_height['result']['height']}")

# Get EVM block number
evm_block_number = send_request("eth_blockNumber", [], "ext/bc/C/rpc")
print(f"EVM Block Number (C): {int(evm_block_number['result'], 16)}") # Hex to int conversion

# Check if chains are bootstrapped
for chain in ["X", "C", "P"]:
    is_bootstrapped = send_request("info.isBootstrapped", {"chain": chain}, "ext/info")
    print(f"Is Chain {chain} Bootstrapped: {is_bootstrapped['result']['isBootstrapped']}")

root@ip-10-0-0-84:~# python3 ~/avax.py
AVM Height (X): 17727
Platform Height (P): 5939765
EVM Block Number (C): 30225793
Is Chain X Bootstrapped: True
Is Chain C Bootstrapped: True
Is Chain P Bootstrapped: True

So with these configuration options, it seems that avalanche C chain is still not syncing from the genesis block #1 since I can't query it:

root@ip-10-0-0-84:~# curl -X POST -H "Content-Type: application/json" \
>   --data '{"method":"eth_getBlockByNumber","params":["0x1",true],"id":1,"jsonrpc":"2.0"}' http://127.0.0.1:8098/ext/bc/C/rpc
{"jsonrpc":"2.0","id":1,"result":null}

the only error I can see is in blockbook:

May 19 15:46:34 ip-10-0-0-84 blockbook[67793]: E0519 15:46:34.817525   67793 sync.go:377] GetBlockHash error height 1: not found
May 19 15:46:35 ip-10-0-0-84 blockbook[67793]: E0519 15:46:35.318798   67793 sync.go:377] GetBlockHash error height 1: not found
May 19 15:46:35 ip-10-0-0-84 blockbook[67793]: E0519 15:46:35.819822   67793 sync.go:377] GetBlockHash error height 1: not found
May 19 15:46:36 ip-10-0-0-84 blockbook[67793]: E0519 15:46:36.321159   67793 sync.go:377] GetBlockHash error height 1: not found
May 19 15:46:36 ip-10-0-0-84 blockbook[67793]: E0519 15:46:36.822513   67793 sync.go:377] GetBlockHash error height 1: not found
May 19 15:46:37 ip-10-0-0-84 blockbook[67793]: E0519 15:46:37.323801   67793 sync.go:377] GetBlockHash error height 1: not found
May 19 15:46:37 ip-10-0-0-84 blockbook[67793]: E0519 15:46:37.825069   67793 sync.go:377] GetBlockHash error height 1: not found

[mattmartinez]

One thing I did notice is 0x0 is retrievable but 0x1 isn't:

root@ip-10-0-0-84:~# curl -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockByNumber","params":["0x1", true]}' http://127.0.0.1:8098/ext/bc/C/rpc | jq .
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": null
}

root@ip-10-0-0-84:~# curl -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockByNumber","params":["0x0", true]}' http://127.0.0.1:8098/ext/bc/C/rpc | jq .
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "blockExtraData": "0x",
    "difficulty": "0x0",
    "extDataHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "extraData": "0x00",
    "gasLimit": "0x5f5e100",
    "gasUsed": "0x0",
    "hash": "0x31ced5b9beb7f8782b014660da0cb18cc409f121f408186886e1ca3e8eeca96b",
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "miner": "0x0000000000000000000000000000000000000000",
    "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "nonce": "0x0000000000000000",
    "number": "0x0",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "size": "0x21c",
    "stateRoot": "0xd65eb1b8604a7aa497d41cd6372663785a5f809a17bd192edb86658ef24e29cc",
    "timestamp": "0x0",
    "totalDifficulty": "0x0",
    "transactions": [],
    "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "uncles": []
  }
}

[matyushkins]

Hello. I have the same. Running just a node does not produce 1,2,3... blocks. The solution to the problem is most likely to use the snapshot. I'll check soon.

[martinboehm]

Hi, FYI, I started Avalanche from snapshot and it worked fine.

[matyushkins]

From the snapshot everything is ok. There are blocks. There are none if you run the node without a snapshot. Apparently cuts the blockchain.

[mattmartinez]

Hi, FYI, I started Avalanche from snapshot and it worked fine.

this seems to work so far, I used the snapshot from http://youravaxsnapshot.senseinode.com

Though I'm having issues with Avalanche now after running it. My C chain log file C.log is flooding with endless lines of:

WARN [06-08|19:38:38.457] <C Chain> github.com/ava-labs/coreth/rpc/handler.go:403: Served debug_traceBlockByHash reqid=3209955 execTime=19.691554ms  procTime=19.691978ms  totalTime=19.692141ms  err="required historical state unavailable (reexec=128)"
WARN [06-08|19:38:38.458] <C Chain> github.com/ava-labs/coreth/rpc/handler.go:403: Served debug_traceBlockByHash reqid=3209954 execTime=20.359601ms  procTime=20.359954ms  totalTime=20.360072ms  err="required historical state unavailable (reexec=128)"
WARN [06-08|19:38:38.460] <C Chain> github.com/ava-labs/coreth/rpc/handler.go:403: Served debug_traceBlockByHash reqid=3209962 execTime=19.434148ms  procTime=19.43445ms   totalTime=19.434582ms  err="required historical state unavailable (reexec=128)"
WARN [06-08|19:38:38.461] <C Chain> github.com/ava-labs/coreth/rpc/handler.go:403: Served debug_traceBlockByHash reqid=3209961 execTime=20.476962ms  procTime=20.47738ms   totalTime=20.477487ms  err="required historical state unavailable (reexec=128)"
WARN [06-08|19:38:38.462] <C Chain> github.com/ava-labs/coreth/rpc/handler.go:403: Served debug_traceBlockByHash reqid=3209964 execTime=19.821707ms  procTime=19.822113ms  totalTime=19.822265ms  err="required historical state unavailable (reexec=128)"

It slowly consumes all the memory on the instance til OOMKiller crashes the server, so I'm restarting Avalanche basically every 1hr to free up memory, not sure whats going on.

[mattmartinez]

It finished syncing and the errors stopped flooding in, so i'll go ahead and close this out since the snapshot ultimately was the fix