Open let4be opened 1 year ago
I am facing same issue
Maybe upgrade to latest?
I'm having the same issue with firehose-eth
In my case, running sync for some time, up to about 1M block, then it suddenly fails but unclear why.
2023-04-03T17:58:54.814Z INFO (reader) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block <empty>", "block_average_parse_time": "0ms/block (over 1min)"}
INFO [04-03|17:58:55.418] Downloader queue stats receiptTasks=0 blockTasks=362 itemSize=231.00B throttle=8192
INFO [04-03|17:58:55.490] Imported new chain segment blocks=1 txs=7 mgas=0.187 elapsed=72.482ms mgasps=2.580 number=956,745 hash=6e5ab3..d27e48 age=7y3mo4d dirty=30.65KiB
INFO [04-03|17:58:55.490] Unindexed transactions blocks=1 txs=0 tail=955,746 elapsed="112.742µs"
2023-04-03T17:58:55.491Z INFO (relayer.src.13010) realtime gate passed {"delta": "62745h1m34.491879003s"}
2023-04-03T17:58:55.491Z INFO (bstream) loading blocks in ForkableHub from one-block-files {"start_block": 956500, "head_block": "#956745 (6e5ab310d4cf79b38f086b029ccc69589974d6bb07f82afd1117d7ed25d27e48)"}
INFO [04-03|17:58:55.894] Imported new chain segment blocks=21 txs=638 mgas=14.810 elapsed=253.664ms mgasps=58.385 number=956,766 hash=e18bf6..d26fab age=7y3mo4d dirty=1.17MiB
INFO [04-03|17:58:55.895] Unindexed transactions blocks=21 txs=25 tail=955,767 elapsed="703.993µs"
INFO [04-03|17:58:56.662] Imported new chain segment blocks=172 txs=595 mgas=24.863 elapsed=764.909ms mgasps=32.505 number=956,938 hash=1af45a..29ddbe age=7y3mo4d dirty=2.80MiB
INFO [04-03|17:58:56.697] Unindexed transactions blocks=172 txs=342 tail=955,939 elapsed=34.972ms
2023-04-03T17:58:56.844Z INFO (index-builder) reading from blocks store: file does not (yet?) exist, retrying in {"filename": "/mnt/ebs/firehome/sf-data/storage/merged-blocks/0000956500.dbin.zst", "base_filename": "0000956500", "retry_delay": "4s"}
2023-04-03T17:58:56.888Z INFO (reader.sub.relayer) reach max buffer size for subcription, closing channel {"capacity": 200}
2023-04-03T17:58:56.890Z INFO (reader.sub.relayer) not pushing block to a closed subscription
2023-04-03T17:58:56.896Z INFO (reader.sub.relayer) not pushing block to a closed subscription
2023-04-03T17:58:56.901Z INFO (reader.sub.relayer) not pushing block to a closed subscription
2023-04-03T17:58:56.904Z INFO (reader.sub.relayer) not pushing block to a closed subscription
.... (repeating logs)....
2023-04-03T17:58:58.635Z INFO (reader.sub.relayer) not pushing block to a closed subscription
2023-04-03T17:58:58.637Z INFO (reader) unsubscribed {"new_length": 0}
2023-04-03T17:58:58.644Z INFO (relayer.src.13010) source shutting down {"endpoint_url": ":13010", "error": "EOF"}
2023-04-03T17:58:58.837Z INFO (relayer) new source factory created
2023-04-03T17:58:58.838Z INFO (relayer.src.13010) starting block source consumption {"endpoint_url": ":13010"}
2023-04-03T17:58:58.838Z INFO (relayer.src.13010) block stream source reading messages {"endpoint_url": ":13010"}
2023-04-03T17:58:58.839Z INFO (reader.sub.relayer) receive block request {"trace_id": "7770c4af8f65cd39e1a028b7ebc439a0", "request": {"requester":"relayer"}}
2023-04-03T17:58:58.839Z INFO (reader.sub.relayer) sending burst {"busrt_size": 0}
2023-04-03T17:58:58.839Z INFO (reader) subscribed {"new_length": 1, "subscriber": "relayer"}
2023-04-03T17:58:58.852Z INFO (relayer.src.13010) realtime gate passed {"delta": "62741h53m19.852322575s"}
2023-04-03T17:58:58.866Z WARN (bstream) too many consecutive unlinkable blocks {"block": "#957418 (db0c8e4907f8a2ffd67b51a0279a14391f7d244157daa07094b3c59f27e9e52e)"}
2023-04-03T17:58:58.866Z ERRO (relayer) unable to process block, shutting down source
2023-04-03T17:58:58.866Z INFO (bstream) reconnecting hub after disconnection. expecting to reconnect and get blocks linking to headnum within delay {"delay": "1m0s", "current_head_block_num": 956954, "error": "too many consecutive unlinkable blocks"}
....
2023-04-03T17:58:58.869Z INFO (relayer) closing block stream server
2023-04-03T17:58:58.870Z ERRO (fireeth)
################################################################
Fatal error in app relayer:
cannot link new blocks to chain after a reconnection
###############################################################
...
2023-04-03T17:58:59.553Z ERRO (derr) dfuse {"error": "cannot link new blocks to chain after a reconnection"}
....
Setup my firehose-eth on AWS using this documentation: https://github.com/streamingfast/firehose-docs/blob/master/firehose-setup/ethereum/local-deployment.md
here are my versions:
[ec2-user@ip-10-0-21-115 firehome]$ ./geth_linux version
INFO [04-03|18:03:13.902] Initializing firehose
INFO [04-03|18:03:13.909] Firehose initialized enabled=false sync_instrumentation_enabled=true mining_enabled=false block_progress_enabled=false compaction_disabled=false archive_blocks_to_keep=0 genesis_provenance="Geth Default" firehose_version=2.0 geth_version=1.10.23-fh2 chain_variant=geth
Geth
Version: 1.10.23-fh2
Git Commit: e7f3686b2f8955a0824140c98f88c4e3c94f9741
Architecture: amd64
Go Version: go1.17.12
Operating System: linux
GOPATH=
GOROOT=go
[ec2-user@ip-10-0-21-115 firehome]$ ./fireeth --version
fireeth version refs/tags/v1.0.0 (Commit 42f870c, Built 2022-09-02T17:05Z)/
not really sure what this error is and how to properly handle it. if I try simply restarting the process, it usually fails with same error but about 1/5 times it does not fail and is able to pickup where it left off and continue processing. but in these cases it fails with same error about 30min later
any idea how I can address this issue ?
Versions you are using for fireeth
and Firehose instrumented geth
are outdated, please use the latest version for both (so https://github.com/streamingfast/firehose-ethereum/releases/tag/v1.3.7 for fireeth
and https://github.com/streamingfast/go-ethereum/releases/tag/geth-v1.11.5-fh2.2 for Firehose instrumented geth
).
Note that a Beacon Client is required for Ethereum Mainnet/Goerli to sync properly.
Please retry with latest version and let us know how it goes.
Looking better now @maoueh after updating latest versions, syncing the beacon client, and getting the fireeth instance to sync with the beacon client. Up to 1.4M blocks and counting. Thanks again for your help here and on Discord @maoueh :)
fireeth version 1.4.10 (Commit 5b548fb, Built 2023-07-27T17:49:35Z)
geth_linux version 1.12.0-fh2-dd3c6c2b
Ctrl-C during chain sync always corrupts (with "cannot link new blocks to chain after a reconnection" and "too many consecutive unlinkable blocks" on next run) and I have to purge and start over. Not expected because the docs say:
Tip: To terminate the Firehose processing and connection to the Ethereum network press the Control + C keys.
The Firehose sync process will shut down gracefully and continue where it left off upon the next restart.
Why firehose process are you terminating? Only the reader? What is your config?
./fireeth -c config.yaml start
start:
args:
- merger
- firehose
- reader-node
- relayer
- combined-index-builder
flags:
log-to-file: false
common-chain-id: "5"
common-network-id: "5"
reader-node-path: /usr/local/bin/geth_linux
reader-node-arguments: "+--goerli --datadir /home/ethereum-goerli-firehose --cache 8192 --maxpeers 100 --metrics --metrics.port 6061 --port=30303 --http.port=8545 --ws.port=8546 --snapshot=true --txlookuplimit=1000"
reader-node-log-to-zap: false
@abourget for planning
Network: BSC Test net Firehose version: v1.3.1 Geth: bsc-v1.1.18-fh2.1-2
Steps to repeat: start syncing, sync some blocks and press ctrl-c or terminate process abruptly(I'm running merger, firehose, reader-node, relayer, combined-index-builder) all in one process. Restart
./fireeth -c config.yaml start
(maybe several times) and watch it die withcannot link new blocks to chain after a reconnection
sometimes it also enters infinite loop spamming console with millions of2023-01-18T09:41:07.186+0400 INFO (reader.sub.relayer) not pushing block to a closed subscription
I think firehose should be able to restart gracefully even after abrupt termination(and it fails to do so even when doing graceful shutdown via ctrl+c)
edit: on yet another restart with the same state it started to complain on this
2023-01-18T09:43:46.926+0400 WARN (bstream) too many consecutive unlinkable blocks {"block": "#136834 (06ce4bb2b6c54d9c)", "consecutive_unlinkable_blocks": 24300, "last_block_sent": "#112408 (444c547f0115596d)"}
in the result it seems that local state is corrupted and I cannot continue syncing data... is there any way to repair it?
config file
example log:https://pastebin.com/w3Rn2k8r