search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.15k stars 1.13k forks source link

2 years later and still randomly stuck blocks, why? #4975

Closed 1blockologist closed 2 years ago

1blockologist commented 2 years ago

Not a bug, no stack trace, can we just have an honest discussion, why does the Erigon client still get stuck on blocks randomly after all this time, all these edge cases found, all these git merges done to fix them, why is that a thing? In what world is completely resyncing a viable solution?

This seems fundamentally flawed

AskAlexSharov commented 2 years ago
Screen Shot 2022-08-10 at 10 07 25

why copier broken? network cable, drivers, electricity socket, paper stuck, ip conflict, update of printer, update of computer, update of router, new firewall on router, color over, and then again broken network cable. and user say "but I just wanna working copier... isn't it small wish?"

Let's search word "stuck": erigon: https://github.com/ledgerwatch/erigon/search?q=stuck&type=issues geth: https://github.com/ethereum/go-ethereum/search?q=stuck&type=issues OE: https://github.com/openethereum/openethereum/search?q=stuck&type=issues Android: https://issuetracker.google.com/issues?q=stuck Seems like everything is stuck :-) Actually not - most of the time everything works fine. But your feedback is extremely important - everybody need to think what we can do to address it.

All bugs are coming from: dev processes, release processes, priorities/focus. We have CI tests: https://github.com/ethereum/tests and https://github.com/ethereum/hive2 , but we don't have much simulation test (where many nodes are attacking each other) - anybody can take leadership in this area and drive it. Our current focus is still on "Ergonomic, size and speed of ArchiveNode" - not on "light node", not on "speed of new blocks processing" - it means other areas will suffer a bit, especially when adding new networks/consensuses/syncmodes support: TheMerge, BSC, Bor/Polygon, Snapshot sync, etc... Very likely that 2 years ago your node had 80% different sync logic.

"completely resyncing a viable solution?" - no, we very rare recommend this solution (of course in our team some people are more radical, some less). I would say: almost never.

My opinion:

Thanks for feedback.

tjayrush commented 2 years ago

My node was stuck yesterday but I let it run overnight. I restarted the node this morning. At first, it appeared to sync, but then seems to have had problems. Here's the log. It says something weird about "Incorrect snapshot enablement" when it first started

INFO[08-11|06:14:39.831] Opening Database                         label=chaindata path=/Users/jrush/Data/erigon-mainnet-devel/chaindata
INFO[08-11|06:14:39.839] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
WARN[08-11|06:14:39.839] Incorrect snapshot enablement            got=true change_to=false
INFO[08-11|06:14:39.839] Effective                                prune_flags= snapshot_flags=
INFO[08-11|06:14:39.839] Initialising Ethereum protocol           network=1
INFO[08-11|06:14:39.839] Disk storage enabled for ethash DAGs     dir=/Users/jrush/Data/erigon-mainnet-devel/ethash-dags count=2
INFO[08-11|06:14:43.469] Starting private RPC server              on=localhost:9091
INFO[08-11|06:14:43.469] new subscription to logs established
INFO[08-11|06:14:43.472] rpc filters: subscribing to Erigon events
INFO[08-11|06:14:43.472] New txs subscriber joined
INFO[08-11|06:14:43.472] new subscription to newHeaders established
INFO[08-11|06:14:43.472] Reading JWT secret                       path=/Users/jrush/Data/erigon-mainnet-devel/jwt.hex
INFO[08-11|06:14:43.473] HTTP endpoint opened for Engine API      url=localhost:8551 ws=true ws.compression=true
INFO[08-11|06:14:43.478] HTTP endpoint opened                     url=localhost:8545 ws=false ws.compression=true grpc=false
INFO[08-11|06:14:43.482] [1/16 Headers] Waiting for headers...    from=15215097
INFO[08-11|06:14:43.502] Started P2P networking                   version=66 self=enode://1163617308fdacf6da34f2722384b248f30e2fdcbdc7cdf11c8af9aeea1c4f31f28c1e858ea62bb43e02b51bbaf53254d8aa209b561187b346ff053b07b501ba@127.0.0.1:30303 name=erigon/v2022.99.99-dev-e702e48b/darwin-arm64/go1.18.4
INFO[08-11|06:15:00.536] [txpool] Started
INFO[08-11|06:15:03.483] [1/16 Headers] Wrote block headers       number=15215097 blk/second=0.000 alloc=110.4MB sys=155.3MB
INFO[08-11|06:15:23.483] [1/16 Headers] Wrote block headers       number=15215097 blk/second=0.000 alloc=239.0MB sys=276.2MB
INFO[08-11|06:15:43.475] [txpool] stat                            block=15215097 pending=4952 baseFee=30000 queued=30000 alloc=290.7MB sys=362.0MB
INFO[08-11|06:15:43.482] [1/16 Headers] Wrote block headers       number=15215097 blk/second=0.000 alloc=290.8MB sys=362.0MB
INFO[08-11|06:16:04.504] [1/16 Headers] Inserting headers         progress=15313946 queue=1
INFO[08-11|06:16:23.485] [1/16 Headers] Inserting headers         progress=15318565 queue=1
INFO[08-11|06:16:29.810] [1/16 Headers] Processed                 highest inserted=15320131 age=17s
INFO[08-11|06:16:29.810] [1/16 Headers] DONE                      in=1m46.328062875s
INFO[08-11|06:16:35.907] [4/16 Bodies] Processing bodies...       from=15215097 to=15320131
WARN[08-11|06:16:37.973] Handling incoming message                stream=RecvMessage err="decode BlockBodiesPacket66: rlp: input list has too many elements for eth.BlockRawBodiesPacket66"
INFO[08-11|06:16:39.839] [p2p] GoodPeers                          eth66=6
INFO[08-11|06:16:43.502] [txpool] stat                            block=15215097 pending=4953 baseFee=30000 queued=30000 alloc=480.1MB sys=870.8MB
INFO[08-11|06:16:55.908] [4/16 Bodies] Wrote block bodies         block_num=15215121 delivery/sec=8.0MB wasted/sec=2.3MB alloc=952.9MB sys=1.0GB
INFO[08-11|06:17:15.908] [4/16 Bodies] Wrote block bodies         block_num=15215145 delivery/sec=10.1MB wasted/sec=4.0MB alloc=910.3MB sys=1.5GB
INFO[08-11|06:17:35.908] [4/16 Bodies] Wrote block bodies         block_num=15215171 delivery/sec=11.0MB wasted/sec=2.8MB alloc=1.6GB sys=1.8GB
INFO[08-11|06:17:43.472] [txpool] stat                            block=15215097 pending=4953 baseFee=30000 queued=30000 alloc=1.4GB sys=2.0GB
INFO[08-11|06:17:55.907] [4/16 Bodies] Wrote block bodies         block_num=15215171 delivery/sec=11.5MB wasted/sec=2.1MB alloc=1.4GB sys=2.2GB
WARN[08-11|06:18:07.389] Handling incoming message                stream=RecvMessage err="decode BlockBodiesPacket66: rlp: input list has too many elements for eth.BlockRawBodiesPacket66"
INFO[08-11|06:18:15.907] [4/16 Bodies] Wrote block bodies         block_num=15215171 delivery/sec=11.6MB wasted/sec=2.4MB alloc=1.8GB sys=2.7GB
INFO[08-11|06:18:35.907] [4/16 Bodies] Wrote block bodies         block_num=15215171 delivery/sec=10.5MB wasted/sec=2.5MB alloc=1.8GB sys=3.2GB
INFO[08-11|06:18:39.841] [p2p] GoodPeers                          eth66=11
INFO[08-11|06:18:43.496] [txpool] stat                            block=15215097 pending=5333 baseFee=30000 queued=30000 alloc=2.3GB sys=3.2GB
INFO[08-11|06:18:55.921] [4/16 Bodies] Wrote block bodies         block_num=15215171 delivery/sec=7.9MB wasted/sec=3.6MB alloc=3.0GB sys=3.2GB
INFO[08-11|06:19:15.907] [4/16 Bodies] Wrote block bodies         block_num=15215200 delivery/sec=8.2MB wasted/sec=5.4MB alloc=2.9GB sys=3.7GB
INFO[08-11|06:19:35.907] [4/16 Bodies] Wrote block bodies         block_num=15215224 delivery/sec=8.3MB wasted/sec=4.9MB alloc=2.4GB sys=4.3GB
WARN[08-11|06:19:42.008] Handling incoming message                stream=RecvMessage err="decode BlockBodiesPacket66: rlp: input list has too many elements for eth.BlockRawBodiesPacket66"
INFO[08-11|06:19:44.576] [txpool] stat                            block=15215097 pending=5334 baseFee=30000 queued=30000 alloc=2.9GB sys=4.3GB
INFO[08-11|06:19:55.908] [4/16 Bodies] Wrote block bodies         block_num=15215224 delivery/sec=7.0MB wasted/sec=4.2MB alloc=3.5GB sys=4.3GB
INFO[08-11|06:20:15.907] [4/16 Bodies] Wrote block bodies         block_num=15215224 delivery/sec=6.8MB wasted/sec=4.5MB alloc=2.6GB sys=4.9GB
INFO[08-11|06:20:35.911] [4/16 Bodies] Wrote block bodies         block_num=15215224 delivery/sec=6.2MB wasted/sec=4.3MB alloc=3.7GB sys=4.9GB
INFO[08-11|06:20:39.838] [p2p] GoodPeers                          eth66=15
INFO[08-11|06:20:43.477] [txpool] stat                            block=15215097 pending=5334 baseFee=30000 queued=30000 alloc=4.1GB sys=4.9GB
INFO[08-11|06:20:55.909] [4/16 Bodies] Wrote block bodies         block_num=15215493 delivery/sec=4.5MB wasted/sec=5.1MB alloc=4.6GB sys=4.9GB
INFO[08-11|06:21:15.907] [4/16 Bodies] Wrote block bodies         block_num=15215493 delivery/sec=5.6MB wasted/sec=5.2MB alloc=3.4GB sys=5.6GB
INFO[08-11|06:21:35.907] [4/16 Bodies] Wrote block bodies         block_num=15215879 delivery/sec=4.1MB wasted/sec=5.8MB alloc=4.4GB sys=5.6GB
INFO[08-11|06:21:43.471] [txpool] stat                            block=15215097 pending=5341 baseFee=30000 queued=30000 alloc=4.7GB sys=5.6GB
INFO[08-11|06:21:55.907] [4/16 Bodies] Wrote block bodies         block_num=15218649 delivery/sec=6.7MB wasted/sec=4.2MB alloc=5.5GB sys=5.8GB
INFO[08-11|06:22:15.907] [4/16 Bodies] Wrote block bodies         block_num=15218649 delivery/sec=5.4MB wasted/sec=4.1MB alloc=3.6GB sys=6.2GB
INFO[08-11|06:22:35.910] [4/16 Bodies] Wrote block bodies         block_num=15218649 delivery/sec=2.9MB wasted/sec=3.5MB alloc=4.2GB sys=6.2GB
WARN[08-11|06:22:36.033] Handling incoming message                stream=RecvMessage err="decode BlockBodiesPacket66: rlp: input list has too many elements for eth.BlockRawBodiesPacket66"
INFO[08-11|06:22:39.838] [p2p] GoodPeers                          eth66=15
INFO[08-11|06:22:43.498] [txpool] stat                            block=15215097 pending=5343 baseFee=30000 queued=30000 alloc=4.6GB sys=6.2GB
INFO[08-11|06:22:55.910] [4/16 Bodies] Wrote block bodies         block_num=15221585 delivery/sec=8.3MB wasted/sec=1.9MB alloc=5.3GB sys=6.2GB
INFO[08-11|06:23:15.907] [4/16 Bodies] Wrote block bodies         block_num=15237349 delivery/sec=9.0MB wasted/sec=1.6MB alloc=3.7GB sys=6.3GB
INFO[08-11|06:23:35.907] [4/16 Bodies] Wrote block bodies         block_num=15237349 delivery/sec=9.0MB wasted/sec=3.3MB alloc=4.9GB sys=6.3GB
INFO[08-11|06:23:43.470] [txpool] stat                            block=15215097 pending=5345 baseFee=30000 queued=30000 alloc=5.4GB sys=6.3GB
INFO[08-11|06:23:55.907] [4/16 Bodies] Wrote block bodies         block_num=15237349 delivery/sec=8.9MB wasted/sec=3.9MB alloc=6.1GB sys=6.5GB
INFO[08-11|06:24:15.907] [4/16 Bodies] Wrote block bodies         block_num=15248703 delivery/sec=11.0MB wasted/sec=3.1MB alloc=3.8GB sys=6.6GB
INFO[08-11|06:24:35.909] [4/16 Bodies] Wrote block bodies         block_num=15248703 delivery/sec=10.7MB wasted/sec=3.5MB alloc=2.3GB sys=6.6GB
INFO[08-11|06:24:39.844] [p2p] GoodPeers                          eth66=17
INFO[08-11|06:24:43.955] [txpool] stat                            block=15215097 pending=5347 baseFee=30000 queued=30000 alloc=2.8GB sys=6.6GB
INFO[08-11|06:24:55.906] [4/16 Bodies] Wrote block bodies         block_num=15248703 delivery/sec=9.1MB wasted/sec=4.6MB alloc=2.2GB sys=6.6GB
INFO[08-11|06:25:15.907] [4/16 Bodies] Wrote block bodies         block_num=15248731 delivery/sec=10.7MB wasted/sec=3.4MB alloc=3.6GB sys=6.6GB
INFO[08-11|06:25:35.907] [4/16 Bodies] Wrote block bodies         block_num=15251711 delivery/sec=9.2MB wasted/sec=4.8MB alloc=3.4GB sys=6.6GB
INFO[08-11|06:25:43.471] [txpool] stat                            block=15215097 pending=5347 baseFee=30000 queued=30000 alloc=3.9GB sys=6.6GB
INFO[08-11|06:25:55.906] [4/16 Bodies] Wrote block bodies         block_num=15251711 delivery/sec=8.5MB wasted/sec=5.1MB alloc=2.6GB sys=6.6GB
INFO[08-11|06:26:15.906] [4/16 Bodies] Wrote block bodies         block_num=15251734 delivery/sec=8.4MB wasted/sec=5.8MB alloc=3.9GB sys=6.6GB
INFO[08-11|06:26:35.906] [4/16 Bodies] Wrote block bodies         block_num=15251856 delivery/sec=8.4MB wasted/sec=5.9MB alloc=3.3GB sys=6.6GB
INFO[08-11|06:26:39.837] [p2p] GoodPeers                          eth66=20
INFO[08-11|06:26:43.476] [txpool] stat                            block=15215097 pending=5347 baseFee=30000 queued=30000 alloc=3.9GB sys=6.6GB
INFO[08-11|06:26:55.906] [4/16 Bodies] Wrote block bodies         block_num=15251875 delivery/sec=8.0MB wasted/sec=6.6MB alloc=4.8GB sys=6.6GB
INFO[08-11|06:27:15.906] [4/16 Bodies] Wrote block bodies         block_num=15251875 delivery/sec=6.6MB wasted/sec=7.5MB alloc=3.8GB sys=6.6GB
INFO[08-11|06:27:35.923] [4/16 Bodies] Wrote block bodies         block_num=15254536 delivery/sec=6.3MB wasted/sec=7.7MB alloc=5.3GB sys=6.6GB
INFO[08-11|06:27:43.476] [txpool] stat                            block=15215097 pending=5355 baseFee=30000 queued=30000 alloc=5.8GB sys=6.6GB
INFO[08-11|06:27:55.906] [4/16 Bodies] Wrote block bodies         block_num=15256254 delivery/sec=8.9MB wasted/sec=5.3MB alloc=3.8GB sys=6.7GB
INFO[08-11|06:28:15.907] [4/16 Bodies] Wrote block bodies         block_num=15266809 delivery/sec=10.8MB wasted/sec=3.1MB alloc=5.3GB sys=6.7GB
INFO[08-11|06:28:35.905] [4/16 Bodies] Wrote block bodies         block_num=15266809 delivery/sec=13.4MB wasted/sec=1.7MB alloc=3.2GB sys=6.7GB
INFO[08-11|06:28:39.837] [p2p] GoodPeers                          eth66=24
INFO[08-11|06:28:43.472] [txpool] stat                            block=15215097 pending=5355 baseFee=30000 queued=30000 alloc=3.8GB sys=6.8GB
INFO[08-11|06:28:55.905] [4/16 Bodies] Wrote block bodies         block_num=15279082 delivery/sec=12.3MB wasted/sec=1.9MB alloc=4.8GB sys=6.8GB
INFO[08-11|06:29:15.905] [4/16 Bodies] Wrote block bodies         block_num=15285341 delivery/sec=13.8MB wasted/sec=1.9MB alloc=3.3GB sys=6.8GB
WARN[08-11|06:29:19.855] Handling incoming message                stream=RecvMessage err="decode BlockBodiesPacket66: rlp: input list has too many elements for eth.BlockRawBodiesPacket66"
WARN[08-11|06:29:22.556] Handling incoming message                stream=RecvMessage err="decode BlockBodiesPacket66: rlp: input list has too many elements for eth.BlockRawBodiesPacket66"
INFO[08-11|06:29:35.906] [4/16 Bodies] Wrote block bodies         block_num=15287210 delivery/sec=12.0MB wasted/sec=2.4MB alloc=2.7GB sys=6.8GB
INFO[08-11|06:29:43.869] [txpool] stat                            block=15215097 pending=5362 baseFee=30000 queued=30000 alloc=1.8GB sys=6.8GB
INFO[08-11|06:29:55.907] [4/16 Bodies] Wrote block bodies         block_num=15287210 delivery/sec=11.5MB wasted/sec=3.6MB alloc=2.7GB sys=6.8GB
INFO[08-11|06:30:15.905] [4/16 Bodies] Wrote block bodies         block_num=15288278 delivery/sec=12.3MB wasted/sec=2.2MB alloc=2.7GB sys=6.8GB
INFO[08-11|06:30:35.908] [4/16 Bodies] Wrote block bodies         block_num=15288278 delivery/sec=8.6MB wasted/sec=3.4MB alloc=2.3GB sys=6.8GB
INFO[08-11|06:30:39.836] [p2p] GoodPeers                          eth66=26
INFO[08-11|06:30:43.472] [txpool] stat                            block=15215097 pending=5362 baseFee=30000 queued=30000 alloc=2.9GB sys=6.8GB
INFO[08-11|06:30:55.914] [4/16 Bodies] Wrote block bodies         block_num=15289851 delivery/sec=10.0MB wasted/sec=5.3MB alloc=3.8GB sys=6.8GB
INFO[08-11|06:31:15.906] [4/16 Bodies] Wrote block bodies         block_num=15289881 delivery/sec=7.7MB wasted/sec=5.0MB alloc=3.0GB sys=6.8GB
INFO[08-11|06:31:35.946] [4/16 Bodies] Wrote block bodies         block_num=15290203 delivery/sec=6.2MB wasted/sec=3.7MB alloc=3.9GB sys=6.8GB
INFO[08-11|06:31:43.470] [txpool] stat                            block=15215097 pending=5380 baseFee=30000 queued=30000 alloc=4.3GB sys=6.8GB
INFO[08-11|06:31:55.905] [4/16 Bodies] Wrote block bodies         block_num=15290203 delivery/sec=6.2MB wasted/sec=5.3MB alloc=2.9GB sys=6.8GB
INFO[08-11|06:32:15.906] [4/16 Bodies] Wrote block bodies         block_num=15290225 delivery/sec=6.7MB wasted/sec=6.7MB alloc=4.2GB sys=6.8GB
INFO[08-11|06:32:39.836] [p2p] GoodPeers                          eth66=27
INFO[08-11|06:32:43.471] [txpool] stat                            block=15215097 pending=5381 baseFee=30000 queued=30000 alloc=3.2GB sys=6.8GB
INFO[08-11|06:32:46.097] [4/16 Bodies] Wrote block bodies         block_num=15297608 delivery/sec=6.5MB wasted/sec=5.2MB alloc=3.3GB sys=6.8GB
INFO[08-11|06:32:55.905] [4/16 Bodies] Wrote block bodies         block_num=15303102 delivery/sec=3.4MB wasted/sec=3.6MB alloc=3.7GB sys=6.8GB
INFO[08-11|06:33:15.913] [4/16 Bodies] Wrote block bodies         block_num=15310310 delivery/sec=1.3MB wasted/sec=971.0KB alloc=4.0GB sys=6.8GB
INFO[08-11|06:33:35.919] [4/16 Bodies] Wrote block bodies         block_num=15318337 delivery/sec=448.6KB wasted/sec=0B alloc=4.2GB sys=6.8GB
INFO[08-11|06:33:43.483] [txpool] stat                            block=15215097 pending=5382 baseFee=30000 queued=30000 alloc=4.2GB sys=6.8GB
INFO[08-11|06:34:07.732] [4/16 Bodies] Processed                  highest=15320131
INFO[08-11|06:34:07.732] [4/16 Bodies] DONE                       in=17m31.812712417s
INFO[08-11|06:34:07.732] [5/16 Senders] Started                   from=15215097 to=15320131
INFO[08-11|06:34:37.739] [5/16 Senders] Recovery                  block_number=15229626
INFO[08-11|06:34:39.853] [p2p] GoodPeers                          eth66=29
INFO[08-11|06:34:46.450] [txpool] stat                            block=15215097 pending=5388 baseFee=30000 queued=30000 alloc=2.5GB sys=7.0GB
INFO[08-11|06:35:07.734] [5/16 Senders] Recovery                  block_number=15246581
INFO[08-11|06:35:37.743] [5/16 Senders] Recovery                  block_number=15262908
INFO[08-11|06:35:43.499] [txpool] stat                            block=15215097 pending=5389 baseFee=30000 queued=30000 alloc=3.2GB sys=7.4GB
INFO[08-11|06:36:07.745] [5/16 Senders] Recovery                  block_number=15277974
INFO[08-11|06:36:13.894] Flushed buffer file                      name=/Users/jrush/Data/erigon-mainnet-devel/etl-temp/erigon-sortable-buf-1832897807 alloc=3.8GB sys=7.4GB
INFO[08-11|06:36:37.741] [5/16 Senders] Recovery                  block_number=15295111
INFO[08-11|06:36:39.858] [p2p] GoodPeers                          eth66=33
INFO[08-11|06:36:43.493] [txpool] stat                            block=15215097 pending=5409 baseFee=30000 queued=30000 alloc=2.7GB sys=7.4GB
INFO[08-11|06:37:07.732] [5/16 Senders] Recovery                  block_number=15312820
INFO[08-11|06:37:20.133] Flushed buffer file                      name=/Users/jrush/Data/erigon-mainnet-devel/etl-temp/erigon-sortable-buf-3870410149 alloc=4.3GB sys=7.4GINFO[08-11|06:14:37.672] torrent verbosity                        level=WRN                                                                                     [139/1840]
INFO[08-11|06:14:39.774] Set global gas cap                       cap=50000000
INFO[08-11|06:14:39.831] Opening Database                         label=chaindata path=/Users/jrush/Data/erigon-mainnet-devel/chaindata
INFO[08-11|06:14:39.839] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Sp
urious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Ar
row Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d
0906a34e69aec8c0db1cb8fa3
WARN[08-11|06:14:39.839] Incorrect snapshot enablement            got=true change_to=false
INFO[08-11|06:14:39.839] Effective                                prune_flags= snapshot_flags=
INFO[08-11|06:14:39.839] Initialising Ethereum protocol           network=1
INFO[08-11|06:14:39.839] Disk storage enabled for ethash DAGs     dir=/Users/jrush/Data/erigon-mainnet-devel/ethash-dags count=2
INFO[08-11|06:14:43.469] Starting private RPC server              on=localhost:9091
INFO[08-11|06:14:43.469] new subscription to logs established
INFO[08-11|06:14:43.472] rpc filters: subscribing to Erigon events
INFO[08-11|06:14:43.472] New txs subscriber joined
INFO[08-11|06:14:43.472] new subscription to newHeaders established
INFO[08-11|06:14:43.472] Reading JWT secret                       path=/Users/jrush/Data/erigon-mainnet-devel/jwt.hex
INFO[08-11|06:14:43.473] HTTP endpoint opened for Engine API      url=localhost:8551 ws=true ws.compression=true
INFO[08-11|06:14:43.478] HTTP endpoint opened                     url=localhost:8545 ws=false ws.compression=true grpc=false
INFO[08-11|06:14:43.482] [1/16 Headers] Waiting for headers...    from=15215097
tjayrush commented 2 years ago

Update: I killed it again, pulled the latest code for devel branch, re-built and restarted, and now it seems to be syncing again. Not sure if this is helpful.

tjayrush commented 2 years ago

Update: The problem returns. The sync got to Stage 5 and then hit this problem:

WARN[08-11|07:52:36.854] [6/16 Execution] Execution failed        block=15215098 hash=0x953feb256c9c3624fcdc2fe2c2b07c2a2d4fca3238368384df55debc10d7cc42 err="could not ap
ply tx 6 from block 15215098 [0x1ed616606beda1eeb757d23fe5893f7f57a5feffdf67726c18c64614318bdbd2]: nonce too high: address 0x5aa40C7C8158D8E29CA480d7E05E5a32dD819332, tx:
 5399 state: 5398"
INFO[08-11|07:52:36.854] UnwindTo                                 block=15215097 bad_block_hash=0x953feb256c9c3624fcdc2fe2c2b07c2a2d4fca3238368384df55debc10d7cc42

and has now seemed to revert to starting where it was earlier stuck.

It now seems stuck in a loop doing:

INFO[08-11|08:09:25.805] Queue sizes                              anchors=1 links=130868 persisted=0
INFO[08-11|08:09:25.805] {15189754-15320617 links=130868 (15189754-15320563-15320578-15320586-15320617 15320617)} => 9fc472e4d8a965da1dd12078683476e6df8a1885ce752adb1048722dcc17eea8, anchorQueue.idx=0, next retry in 4.981825292s
INFO[08-11|08:09:45.800] [1/16 Headers] Wrote block headers       number=15215097 blk/second=0.000 alloc=615.3MB sys=5.5GB
INFO[08-11|08:09:45.800] Req/resp stats                           req=2043 reqMin=15187858 reqMax=15189752 skel=20 skelMin=15216633 skelMax=15511353 resp=2065 respMin=15187867 respMax=15319545 dups=40916
AskAlexSharov commented 2 years ago

let's close this issue because no activity related to main topi. if you have some sync problems, please create github issue - we will investigate. thank you.