search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.12k stars 1.1k forks source link

Caplin Antiqate: panic on reading data while indexing #11642

Closed AskAlexSharov closed 3 weeks ago

AskAlexSharov commented 1 month ago

today's main during initial sync of holesky --caplin.backfilling --caplin.archive --caplin.backfilling.blob:

[DBUG] [08-16|11:35:44.961] [index] calculating                      file=v1-001900-002000-beaconblocks.idx
[DBUG] [08-16|11:35:45.806] [index] write                            file=v1-001900-002000-beaconblocks.idx
[DBUG] [08-16|11:35:45.908] Generating idx for beaconblocks          app=caplin progress=0
[DBUG] [08-16|11:35:49.420] Generating idx for beaconblocks          app=caplin progress=20000
[DBUG] [08-16|11:35:50.052] Received block via gossip                slot=2325479
[DBUG] [08-16|11:35:50.052] Block scheduled for later processing     slot=2325479 block=2144045
[INFO] [08-16|11:35:50.337] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=2305859 blockNumber=2126311 frozenBlocks=2122999 blk/sec=38.4 snapshots=0
[DBUG] [08-16|11:35:53.123] Generating idx for beaconblocks          app=caplin progress=40000
[DBUG] [08-16|11:35:56.797] Generating idx for beaconblocks          app=caplin progress=60000
[DBUG] [08-16|11:36:01.715] Received blob sidecar via gossip         index=0 size=128KB
[DBUG] [08-16|11:36:01.855] Received block via gossip                slot=2325480
[DBUG] [08-16|11:36:01.855] Block scheduled for later processing     slot=2325480 block=2144046
[DBUG] [08-16|11:36:01.995] Generating idx for beaconblocks          app=caplin progress=80000
[DBUG] [08-16|11:36:03.329] [index] created                          app=caplin file=/mnt/erigon/snapshots/v1-002000-002100-beaconblocks.idx.tmp
[DBUG] [08-16|11:36:03.329] [index] calculating                      file=v1-002000-002100-beaconblocks.idx
[DBUG] [08-16|11:36:04.339] [index] write                            file=v1-002000-002100-beaconblocks.idx
[DBUG] [08-16|11:36:04.446] Generating idx for beaconblocks          app=caplin progress=0
[DBUG] [08-16|11:36:08.394] Generating idx for beaconblocks          app=caplin progress=20000
[DBUG] [08-16|11:36:12.515] Generating idx for beaconblocks          app=caplin progress=40000
[DBUG] [08-16|11:36:13.011] Received blob sidecar via gossip         index=0 size=128KB
[DBUG] [08-16|11:36:13.033] Received block via gossip                slot=2325481
[DBUG] [08-16|11:36:13.033] Block scheduled for later processing     slot=2325481 block=2144047
[DBUG] [08-16|11:36:13.044] Received blob sidecar via gossip         index=1 size=128KB
[DBUG] [08-16|11:36:13.479] [p2p] Discovery table                    protocol=any version=v5 len=149 live=58 unsol=0 ips=213 db=0 reval=149 RPC timeout=55
[INFO] [08-16|11:36:13.487] P2P                                      app=caplin peers=65
[DBUG] [08-16|11:36:14.005] [p2p] Discovery table                    protocol=67 version=v4 len=172 live=53 unsol=500 ips=142 db=0 reval=154 RPC timeout=62 unsolicited reply=3836 unknown node=7 expired=82
[DBUG] [08-16|11:36:14.006] [p2p] Dial scheduler                     protocol=67 peers=0/33 tried=3570 static=0 i/o timeout=7 connect: connection reset by peer=145 connect: connection refused=1
[DBUG] [08-16|11:36:14.006] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=1262 EOF=1278 closed by remote=795 i/o timeout=2
[DBUG] [08-16|11:36:14.023] [p2p] Discovery table                    protocol=68 version=v4 len=118 live=41 unsol=500 ips=69 db=0 reval=157 RPC timeout=61 unknown node=6 unsolicited reply=4066 expired=102
[DBUG] [08-16|11:36:14.025] [p2p] Dial scheduler                     protocol=68 peers=0/33 tried=3458 static=0 i/o timeout=2 connect: network is unreachable=1 connect: connection refused=5 connect: connection reset by peer=90 connect: no route to host=1
[DBUG] [08-16|11:36:14.025] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 i/o timeout=3 EOF=1360 too many peers=1237 closed by remote=685 ecies: invalid message=1
[DBUG] [08-16|11:36:16.310] Generating idx for beaconblocks          app=caplin progress=60000
[INFO] [08-16|11:36:20.336] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=2304454 blockNumber=2125001 frozenBlocks=2122999 blk/sec=46.8 snapshots=0
[DBUG] [08-16|11:36:21.082] Generating idx for beaconblocks          app=caplin progress=80000
[DBUG] [08-16|11:36:22.345] [index] created                          app=caplin file=/mnt/erigon/snapshots/v1-002100-002200-beaconblocks.idx.tmp
[DBUG] [08-16|11:36:22.345] [index] calculating                      file=v1-002100-002200-beaconblocks.idx
[DBUG] [08-16|11:36:23.105] [index] write                            file=v1-002100-002200-beaconblocks.idx
[INFO] [08-16|11:36:23.502] [Antiquary] Stopping Caplin to process historical indicies from=0 to=2199999
[DBUG] [08-16|11:36:25.364] Received block via gossip                slot=2325482
[DBUG] [08-16|11:36:25.364] Block scheduled for later processing     slot=2325482 block=2144048
panic: runtime error: index out of range [3532] with length 3532 [recovered]
    panic: ReadHeader(99999), runtime error: index out of range [3532] with length 3532, [caplin_snapshots.go:605 panic.go:770 panic.go:120 elias_fano.go:176 elias_fano.go:187 index.go:364 caplin_snapshots.go:624 antiquary.go:165 run.go:377 asm_amd64.s:1695]

goroutine 53226 [running]:
github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*CaplinSnapshots).ReadHeader.func1()
    github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks/caplin_snapshots.go:605 +0xe8
panic({0x283fce0?, 0xc2fb6f21b0?})
    runtime/panic.go:770 +0x132
github.com/erigontech/erigon-lib/recsplit/eliasfano32.(*EliasFano).get(0xc030c1d560?, 0x1008aae52db?)
    github.com/erigontech/erigon-lib@v0.0.0-00010101000000-000000000000/recsplit/eliasfano32/elias_fano.go:176 +0x265
github.com/erigontech/erigon-lib/recsplit/eliasfano32.(*EliasFano).Get(...)
    github.com/erigontech/erigon-lib@v0.0.0-00010101000000-000000000000/recsplit/eliasfano32/elias_fano.go:187
github.com/erigontech/erigon-lib/recsplit.(*Index).OrdinalLookup(...)
    github.com/erigontech/erigon-lib@v0.0.0-00010101000000-000000000000/recsplit/index.go:364
github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*CaplinSnapshots).ReadHeader(0xc030c1d560, 0x1869f)
    github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks/caplin_snapshots.go:624 +0x217
github.com/erigontech/erigon/cl/antiquary.(*Antiquary).Loop(0xc035972e00)
    github.com/erigontech/erigon/cl/antiquary/antiquary.go:165 +0x86a
github.com/erigontech/erigon/cmd/caplin/caplin1.RunCaplinService.func4()
    github.com/erigontech/erigon/cmd/caplin/caplin1/run.go:377 +0x2d
created by github.com/erigontech/erigon/cmd/caplin/caplin1.RunCaplinService in goroutine 5150
    github.com/erigontech/erigon/cmd/caplin/caplin1/run.go:376 +0x2d15
Giulio2002 commented 1 month ago

GOD THIS IS ANNOYING

Giulio2002 commented 3 weeks ago

this is fixed but you need a fresh resync :( or just do rm <datadir>/snapshots/*beaconblocks* <datadir>/snapshots/*blobsidecar* and restarrt