[e3] Node apparently got corrupted

Using main from 27/07, commit 4376f9151899586de1021131e4c6614e6bf90884

I stopped my fully functional e3 sepolia node yesterday, and today after start it seems it got corrupted somehow (?).

Full logs from yesterday stop + today start:

INFO[07-27|06:53:28.648] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=155ms
INFO[07-27|06:53:52.106] head updated                             head=0x7c6e18a63f17b3587c182cd71f9bbc9b2b2d59ae09ff6dfe155845c56ac29f77 hash=0x7c6e18a63f17b3587c182cd71f9bbc9b2b2d59ae09ff6dfe155845c56ac29f77 number=6384431 execution=817.83065ms mgas/s=34.08 commit=565.328µs alloc=1.7GB sys=7.0GB
INFO[07-27|06:53:53.001] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=172ms
INFO[07-27|06:54:02.764] head updated                             head=0x0c085d95debf38715d663d32dfd68c7f15b35174c5143f22c0f694f750c16003 hash=0x0c085d95debf38715d663d32dfd68c7f15b35174c5143f22c0f694f750c16003 number=6384432 execution=484.041529ms mgas/s=29.84 commit=332.159µs alloc=1.8GB sys=7.0GB
INFO[07-27|06:54:03.384] P2P                                      app=caplin peers=35
INFO[07-27|06:54:03.582] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=152ms
INFO[07-27|06:54:15.322] head updated                             head=0x2653932cbafc1ee03f26a7eac6b4469f8503b881537899f85f07cf221386a427 hash=0x2653932cbafc1ee03f26a7eac6b4469f8503b881537899f85f07cf221386a427 number=6384433 execution=521.370608ms mgas/s=26.77 commit=269.564µs alloc=2.0GB sys=7.0GB
INFO[07-27|06:54:16.104] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=143ms
INFO[07-27|06:54:27.659] head updated                             head=0x490d472938def684553e5f559b53f395e1da080989c1daa4c3092a79cd5530ed hash=0x490d472938def684553e5f559b53f395e1da080989c1daa4c3092a79cd5530ed number=6384434 execution=481.380996ms mgas/s=61.09 commit=839.754µs alloc=2.2GB sys=7.0GB
INFO[07-27|06:54:28.495] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=162ms
INFO[07-27|06:54:38.286] head updated                             head=0x58d7e3dc0df1bf881e7afe43df9c0eae7c30ee2c863a711d9f15fe13b080824f hash=0x58d7e3dc0df1bf881e7afe43df9c0eae7c30ee2c863a711d9f15fe13b080824f number=6384435 execution=284.204183ms mgas/s=32.47 commit=526.772µs alloc=2.2GB sys=7.0GB
^CINFO[07-27|06:54:38.436] Got interrupt, shutting down...          sig=interrupt
INFO[07-27|06:54:38.436] Got interrupt, shutting down...
EROR[07-27|06:54:38.441] runPostForkchoiceInBackground            error="[5/6 TxLookup] prune TxLookUp: loadIntoTable BlockTransactionLookup: stopped"
INFO[07-27|06:54:38.441] Exiting Engine...
INFO[07-27|06:54:38.441] Exiting...
INFO[07-27|06:54:38.442] HTTP endpoint closed                     url=[::]:8545
INFO[07-27|06:54:38.442] HTTP endpoint closed                     url=[::]:8546
INFO[07-27|06:54:38.441] RPC server shutting down
INFO[07-27|06:54:38.442] RPC server shutting down
INFO[07-27|06:54:38.443] RPC server shutting down
INFO[07-27|06:54:38.443] Engine HTTP endpoint close               url=127.0.0.1:8551

INFO[07-28|15:11:35.104] logging to file system                   log dir=/Volumes/erigon3-sepolia/erigon/logs file prefix=erigon log level=info json=false
INFO[07-28|15:11:35.193] Build info                               git_branch=main git_tag=v3.0.0-alpha1-25-g4376f91518 git_commit=4376f9151899586de1021131e4c6614e6bf90884
INFO[07-28|15:11:35.193]
        ########b          oo                               d####b.
        ##                                                      '##
        ##aaaa    ##d###b. dP .d####b. .d####b. ##d###b.     aaad#'
        ##        ##'  '## ## ##'  '## ##'  '## ##'  '##        '##
        ##        ##       ## ##.  .## ##.  .## ##    ##        .##
        ########P dP       dP '####P## '#####P' dP    dP    d#####P
                                   .##
                               d####P

INFO[07-28|15:11:35.193] Starting Erigon on Sepolia testnet...
INFO[07-28|15:11:35.203] Maximum peer count                       ETH=100 total=100
INFO[07-28|15:11:35.203] starting HTTP APIs                       port=8545 APIs=eth,net,erigon,ots
INFO[07-28|15:11:35.203] torrent verbosity                        level=WRN
INFO[07-28|15:11:37.209] Set global gas cap                       cap=50000000
INFO[07-28|15:11:37.210] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=128mb upload.rate=4mb
INFO[07-28|15:11:37.287] Opening Database                         label=chaindata path=/Volumes/erigon3-sepolia/erigon/chaindata
INFO[07-28|15:11:37.296] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
INFO[07-28|15:11:37.312] Initialised chain configuration          config="{ChainID: 11155111, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 0, Muir Glacier: 0, Berlin: 0, London: 0, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: 17000000000000000, Merge Netsplit: 1735371, Shanghai: 1677557088, Cancun: 1706655072, Prague: <nil>, Osaka: <nil>, Engine: ethash}" genesis=0x25a5cc106eea7138acab33231d7160d69cb777ee0c2c553fcddf5138993e6dd9
INFO[07-28|15:14:37.537] Initialising Ethereum protocol           network=11155111
INFO[07-28|15:14:37.538] Disk storage enabled for ethash DAGs     dir=/Volumes/erigon3-sepolia/erigon/ethash-dags count=2
INFO[07-28|15:14:37.770] Starting private RPC server              on=127.0.0.1:9090
INFO[07-28|15:14:37.770] new subscription to logs established
INFO[07-28|15:14:37.773] [Checkpoint Sync] Requesting beacon state uri=https://checkpoint-sync.sepolia.ethpandaops.io/eth/v2/debug/beacon/states/finalized
INFO[07-28|15:14:37.781] [txpool] Started
INFO[07-28|15:14:40.250] Starting caplin
INFO[07-28|15:14:40.251] rpc filters: subscribing to Erigon events
INFO[07-28|15:14:40.253] new subscription to newHeaders established
INFO[07-28|15:14:40.253] New txs subscriber joined
INFO[07-28|15:14:40.259] Reading JWT secret                       path=/Volumes/erigon3-sepolia/erigon/jwt.hex
INFO[07-28|15:14:40.261] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
INFO[07-28|15:14:40.261] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=[::]:8546 http.url=[::]:8545
INFO[07-28|15:14:40.290] [1/6 Snapshots] Requesting downloads
WARN[07-28|15:14:40.303] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
WARN[07-28|15:14:40.303] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
INFO[07-28|15:14:40.332] Started P2P networking                   version=68 self=enode://5536dc6397b676671bd2095bc042e5d9b3a30271f8d9e009f547bc4efdac7307e09f3ecafcbf94f9a0bb8fa80537ac7716206aa2f2a76201c45013afd87112b0@127.0.0.1:30303 name=erigon/v3.00.0-alpha1-4376f915/darwin-amd64/go1.22.5
INFO[07-28|15:14:40.332] Started P2P networking                   version=67 self=enode://5536dc6397b676671bd2095bc042e5d9b3a30271f8d9e009f547bc4efdac7307e09f3ecafcbf94f9a0bb8fa80537ac7716206aa2f2a76201c45013afd87112b0@127.0.0.1:30304 name=erigon/v3.00.0-alpha1-4376f915/darwin-amd64/go1.22.5
INFO[07-28|15:14:40.407] Static peers                             len=0
INFO[07-28|15:14:40.410] [Sentinel] Sentinel started              app=caplin enr=enr:-Ly4QBz5bW3Zw9USLgG75x8TUHsUELY0v_Le9lqw-FXuvgRzOO95UqXskfu6PQMtbD4eCYBa62AAjbaVov9ythS6FeMBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDTH2GRAAAAAP__________gmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQK6qG8VmpDoommEFg6l28TAmV1kliq-1GteiLkQeOA8IYhzeW5jbmV0cwCDdGNwgg-hg3VkcIIPoA
INFO[07-28|15:14:40.413] Started Ethereum 2.0 Gossip Service      app=caplin
INFO[07-28|15:14:40.460] [Caplin] starting clstages loop          app=caplin
INFO[07-28|15:14:40.461] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=5537984
meta_unsteady:10335 wipe txn #3, meta 2
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/sync_committee_contribution_and_proof/ssz_snappy
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/beacon_block/ssz_snappy
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_0/ssz_snappy
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/attester_slashing/ssz_snappy
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_5/ssz_snappy
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/bls_to_execution_change/ssz_snappy
INFO[07-28|15:14:41.408] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_4/ssz_snappy
INFO[07-28|15:14:41.409] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/proposer_slashing/ssz_snappy
INFO[07-28|15:14:41.409] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_1/ssz_snappy
INFO[07-28|15:14:41.409] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_3/ssz_snappy
INFO[07-28|15:14:41.409] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_2/ssz_snappy
INFO[07-28|15:14:41.409] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/beacon_aggregate_and_proof/ssz_snappy
INFO[07-28|15:14:51.122] [1/6 Snapshots] Requesting downloads
WARN[07-28|15:15:01.385] [1/6 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=6359999 min_in_db=6380000
INFO[07-28|15:15:01.388] [snapshots:download] Stat                blocks=6360k indices=6360k alloc=1.4GB sys=2.4GB
INFO[07-28|15:15:01.411] [snapshots:history] Stat                 blocks=6373k txs=276m txNum2blockNum="128=5623K,160=6107K,176=6359K,177=6373K" first_history_idx_in_db=6373457 last_comitment_block=6373457 last_comitment_tx_num=276562500 alloc=1.4GB sys=2.4GB
INFO[07-28|15:15:02.024] Missing blocks download finished (note: this does not mean that the history is complete, only that the missing blocks need for sync have been downloaded) app=caplin stage=DownloadHistoricalBlocks
INFO[07-28|15:15:05.466] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
INFO[07-28|15:15:05.474] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=5537984 to=5538075
WARN[07-28|15:15:21.869] failed to insert blocks                  err="parent's total difficulty not found with hash d77cc946061f93346961fdf47cc73621d318fc0b1bb4ed030df6e779e3d770e6 and height 6392551: <nil>"
WARN[07-28|15:15:22.011] Could not set forkchoice                 app=caplin stage=ForkChoice err="execution Client RPC failed to retrieve ForkChoiceUpdate response, err: forkchoice: block d32c6872c070884de81e6712e7ff772630339c0501829042ea03ae1c0c6c01f3 not found or was marked invalid"
WARN[07-28|15:15:22.012] error executing clstage                  app=caplin stage=ForkChoice err="execution Client RPC failed to retrieve ForkChoiceUpdate response, err: forkchoice: block d32c6872c070884de81e6712e7ff772630339c0501829042ea03ae1c0c6c01f3 not found or was marked invalid"
INFO[07-28|15:15:40.414] P2P                                      app=caplin peers=5
INFO[07-28|15:16:40.414] P2P                                      app=caplin peers=4
WARN[07-28|15:17:37.526] [p2p] No GoodPeers
INFO[07-28|15:17:37.527] [mem] memory stats                       alloc=1.8GB sys=3.4GB
INFO[07-28|15:17:37.773] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=1.8GB sys=3.4GB
INFO[07-28|15:17:40.414] P2P                                      app=caplin peers=7
INFO[07-28|15:18:40.414] P2P                                      app=caplin peers=13
INFO[07-28|15:19:40.414] P2P                                      app=caplin peers=8
WARN[07-28|15:20:37.526] [p2p] No GoodPeers
INFO[07-28|15:20:37.527] [mem] memory stats                       alloc=2.1GB sys=3.6GB
INFO[07-28|15:20:37.773] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.2GB sys=3.6GB
INFO[07-28|15:20:40.414] P2P                                      app=caplin peers=21
INFO[07-28|15:21:40.414] P2P                                      app=caplin peers=16
INFO[07-28|15:22:30.006] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=5538016 to=5538112
INFO[07-28|15:22:40.414] P2P                                      app=caplin peers=7
INFO[07-28|15:23:01.263] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=5538052 distance-from-chain-tip=12m0s estimated-time-remaining=50s
WARN[07-28|15:23:18.437] failed to insert blocks                  err="parent's total difficulty not found with hash 3f9d3ccde3f97fbab0f59d0c1eb52974b4e20494a2cdf06b97ec17fdbb607b55 and height 6392740: <nil>"
WARN[07-28|15:23:18.540] Could not set forkchoice                 app=caplin stage=ForkChoice err="execution Client RPC failed to retrieve ForkChoiceUpdate response, err: forkchoice: block 9b6ce34de33f3bd570e79bb8cf74a071f919d08aaf69107d6a7fc9fa9a3769a8 not found or was marked invalid"
WARN[07-28|15:23:18.540] error executing clstage                  app=caplin stage=ForkChoice err="execution Client RPC failed to retrieve ForkChoiceUpdate response, err: forkchoice: block 9b6ce34de33f3bd570e79bb8cf74a071f919d08aaf69107d6a7fc9fa9a3769a8 not found or was marked invalid"
WARN[07-28|15:23:37.526] [p2p] No GoodPeers
INFO[07-28|15:23:37.527] [mem] memory stats                       alloc=1.8GB sys=3.7GB
INFO[07-28|15:23:37.774] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=1.9GB sys=3.7GB
INFO[07-28|15:23:40.414] P2P                                      app=caplin peers=6
INFO[07-28|15:24:40.414] P2P                                      app=caplin peers=12
INFO[07-28|15:25:40.414] P2P                                      app=caplin peers=14
WARN[07-28|15:26:37.526] [p2p] No GoodPeers
INFO[07-28|15:26:37.527] [mem] memory stats                       alloc=2.2GB sys=3.7GB
INFO[07-28|15:26:37.774] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.3GB sys=3.7GB
INFO[07-28|15:26:40.414] P2P                                      app=caplin peers=13
INFO[07-28|15:27:40.415] P2P                                      app=caplin peers=24
INFO[07-28|15:28:40.415] P2P                                      app=caplin peers=9
WARN[07-28|15:29:37.527] [p2p] No GoodPeers
INFO[07-28|15:29:37.527] [mem] memory stats                       alloc=2.5GB sys=3.7GB
INFO[07-28|15:29:37.774] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.5GB sys=3.7GB
INFO[07-28|15:29:40.415] P2P                                      app=caplin peers=13
INFO[07-28|15:30:40.415] P2P                                      app=caplin peers=12
INFO[07-28|15:31:40.415] P2P                                      app=caplin peers=11
WARN[07-28|15:32:37.527] [p2p] No GoodPeers
INFO[07-28|15:32:37.528] [mem] memory stats                       alloc=1.8GB sys=3.8GB
INFO[07-28|15:32:37.774] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=1.8GB sys=3.8GB
INFO[07-28|15:32:40.415] P2P                                      app=caplin peers=3
INFO[07-28|15:33:40.415] P2P                                      app=caplin peers=1
INFO[07-28|15:34:40.415] P2P                                      app=caplin peers=1
INFO[07-28|15:35:30.727] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=5538048 to=5538177
WARN[07-28|15:35:37.527] [p2p] No GoodPeers
INFO[07-28|15:35:37.528] [mem] memory stats                       alloc=2.8GB sys=4.0GB
INFO[07-28|15:35:37.775] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.9GB sys=4.0GB
INFO[07-28|15:35:40.415] P2P                                      app=caplin peers=13
INFO[07-28|15:36:02.600] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=5538127 distance-from-chain-tip=10m0s estimated-time-remaining=18s
INFO[07-28|15:36:32.829] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=5538178 distance-from-chain-tip=-12s estimated-time-remaining=0s
WARN[07-28|15:36:33.054] failed to insert blocks                  err="parent's total difficulty not found with hash 19f56fdb1887d44f6f562ee6e7acb4d889c5c036479b772add2a26e9e7a20586 and height 6392767: <nil>"
INFO[07-28|15:36:40.415] P2P                                      app=caplin peers=26
INFO[07-28|15:37:40.415] P2P                                      app=caplin peers=8
WARN[07-28|15:38:37.527] [p2p] No GoodPeers
INFO[07-28|15:38:37.528] [mem] memory stats                       alloc=2.4GB sys=4.0GB
INFO[07-28|15:38:37.775] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.5GB sys=4.0GB
INFO[07-28|15:38:40.415] P2P                                      app=caplin peers=19
INFO[07-28|15:39:40.415] P2P                                      app=caplin peers=5
INFO[07-28|15:40:40.415] P2P                                      app=caplin peers=14
<keeps repeating same logs>

Seems 2 issues:

node somehow go gap in blocks (maybe pruned more than need)
Caplin has feature to repair such gaps - but seems it didn't succeed

I have an idea - how it can happen:

we added concept of "header chain download" and this feature seems started to track progress of block files by headers.seg instead of transactions.seg - And this is reason why blocks was pruned too early (before transactions.seg created and indexed). Probably easy to reproduce.

number=6384434 ->

WARN[07-28|15:15:01.385] [1/6 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=6359999 min_in_db=6380000
INFO[07-28|15:15:01.388] [snapshots:download] Stat                blocks=6360k indices=6360k alloc=1.4GB sys=2.4GB
INFO[07-28|15:15:01.411] [snapshots:history] Stat                 blocks=6373k txs=276m txNum2blockNum="128=5623K,160=6107K,176=6359K,177=6373K" first_history_idx_in_db=6373457 last_comitment_block=6373457 last_comitment_tx_num=276562500 alloc=1.4GB sys=2.4GB
INFO[07-28|15:15:02.024] Missing blocks download finished (note: this does not mean that the history is complete, only that the missing blocks need for sync have been downloaded) app=caplin stage=DownloadHistoricalBlocks
INFO[07-28|15:15:05.466] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
INFO[07-28|15:15:05.474] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=5537984 to=5538075
WARN[07-28|15:15:21.869] failed to insert blocks                  err="parent's total difficulty not found with hash d77cc946061f93346961fdf47cc73621d318fc0b1bb4ed030df6e779e3d770e6 and height 6392551: <nil>"

@wmitsuda show ls datadir/snapshots/

here:

total 528632904
drwxr--r--@ 52 wmitsuda  staff   1.6K Jul 25 15:30 accessor
drwxr--r--@ 54 wmitsuda  staff   1.7K Jul 25 15:29 domain
drwxr--r--@ 23 wmitsuda  staff   736B Jul 25 15:28 history
drwxr--r--@ 51 wmitsuda  staff   1.6K Jul 25 15:30 idx
-rw-r--r--@  1 wmitsuda  staff    60B Jul 21 01:04 prohibit_new_downloads.lock
-rwxr-xr-x@  1 wmitsuda  staff     4B Jul 21 01:04 salt-blocks.txt
-rwxr-xr-x@  1 wmitsuda  staff     4B Jul 20 14:56 salt-state.txt
-rw-r--r--@  1 wmitsuda  staff   1.9M Jul 21 01:04 v1-000000-000500-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   6.6M Jul 25 09:15 v1-000000-000500-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   465B Jul 20 14:56 v1-000000-000500-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-000000-000500-headers.idx
-rw-r--r--@  1 wmitsuda  staff    64M Jul 25 09:15 v1-000000-000500-headers.seg
-rw-r--r--@  1 wmitsuda  staff   1.0K Jul 20 14:56 v1-000000-000500-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   3.1M Jul 21 01:05 v1-000000-000500-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   4.3M Jul 21 01:05 v1-000000-000500-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   1.0M Jul 25 09:15 v1-000000-000500-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   411B Jul 20 15:00 v1-000000-000500-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   1.9M Jul 21 01:04 v1-000500-001000-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   5.6M Jul 25 09:15 v1-000500-001000-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   445B Jul 20 14:56 v1-000500-001000-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-000500-001000-headers.idx
-rw-r--r--@  1 wmitsuda  staff    64M Jul 25 09:15 v1-000500-001000-headers.seg
-rw-r--r--@  1 wmitsuda  staff   1.0K Jul 20 14:56 v1-000500-001000-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   3.2M Jul 21 01:05 v1-000500-001000-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   4.8M Jul 21 01:05 v1-000500-001000-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   5.3M Jul 25 09:15 v1-000500-001000-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   451B Jul 20 15:00 v1-000500-001000-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   1.9M Jul 21 01:04 v1-001000-001500-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   7.8M Jul 25 09:15 v1-001000-001500-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   465B Jul 20 14:56 v1-001000-001500-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-001000-001500-headers.idx
-rw-r--r--@  1 wmitsuda  staff    78M Jul 25 09:15 v1-001000-001500-headers.seg
-rw-r--r--@  1 wmitsuda  staff   1.2K Jul 20 14:56 v1-001000-001500-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   5.4M Jul 21 01:05 v1-001000-001500-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    10M Jul 21 01:05 v1-001000-001500-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   111M Jul 25 09:15 v1-001000-001500-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   1.5K Jul 20 15:00 v1-001000-001500-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   1.8M Jul 21 01:04 v1-001500-002000-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   3.8M Jul 25 09:15 v1-001500-002000-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   425B Jul 20 14:56 v1-001500-002000-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-001500-002000-headers.idx
-rw-r--r--@  1 wmitsuda  staff    89M Jul 25 09:15 v1-001500-002000-headers.seg
-rw-r--r--@  1 wmitsuda  staff   1.3K Jul 20 14:56 v1-001500-002000-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   9.0M Jul 21 01:05 v1-001500-002000-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    18M Jul 21 01:05 v1-001500-002000-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   420M Jul 25 09:15 v1-001500-002000-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   4.5K Jul 20 15:00 v1-001500-002000-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   1.8M Jul 21 01:04 v1-002000-002500-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   3.8M Jul 25 09:15 v1-002000-002500-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   425B Jul 20 14:56 v1-002000-002500-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-002000-002500-headers.idx
-rw-r--r--@  1 wmitsuda  staff    90M Jul 25 09:15 v1-002000-002500-headers.seg
-rw-r--r--@  1 wmitsuda  staff   1.3K Jul 20 14:56 v1-002000-002500-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   7.9M Jul 21 01:05 v1-002000-002500-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    16M Jul 21 01:05 v1-002000-002500-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   501M Jul 25 09:15 v1-002000-002500-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   5.3K Jul 20 15:00 v1-002000-002500-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   1.8M Jul 21 01:04 v1-002500-003000-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   4.5M Jul 25 09:15 v1-002500-003000-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   445B Jul 20 14:56 v1-002500-003000-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-002500-003000-headers.idx
-rw-r--r--@  1 wmitsuda  staff    96M Jul 25 09:15 v1-002500-003000-headers.seg
-rw-r--r--@  1 wmitsuda  staff   1.3K Jul 20 14:56 v1-002500-003000-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   8.1M Jul 21 01:06 v1-002500-003000-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    16M Jul 21 01:05 v1-002500-003000-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   505M Jul 25 09:15 v1-002500-003000-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   5.3K Jul 20 15:00 v1-002500-003000-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.5M Jul 21 01:04 v1-003000-003500-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    56M Jul 25 09:15 v1-003000-003500-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   947B Jul 20 14:56 v1-003000-003500-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-003000-003500-headers.idx
-rw-r--r--@  1 wmitsuda  staff   162M Jul 25 09:15 v1-003000-003500-headers.seg
-rw-r--r--@  1 wmitsuda  staff   2.0K Jul 20 14:56 v1-003000-003500-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    39M Jul 21 01:09 v1-003000-003500-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    92M Jul 21 01:08 v1-003000-003500-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   6.9G Jul 25 09:15 v1-003000-003500-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    69K Jul 20 15:00 v1-003000-003500-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.6M Jul 21 01:04 v1-003500-004000-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    62M Jul 25 09:15 v1-003500-004000-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   1.0K Jul 20 14:56 v1-003500-004000-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   2.7M Jul 21 01:04 v1-003500-004000-headers.idx
-rw-r--r--@  1 wmitsuda  staff   198M Jul 25 09:15 v1-003500-004000-headers.seg
-rw-r--r--@  1 wmitsuda  staff   2.3K Jul 20 14:56 v1-003500-004000-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    82M Jul 21 01:18 v1-003500-004000-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   198M Jul 21 01:16 v1-003500-004000-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    29G Jul 25 09:15 v1-003500-004000-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   286K Jul 20 15:00 v1-003500-004000-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-004000-004100-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-004000-004100-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004000-004100-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   545K Jul 21 01:04 v1-004000-004100-headers.idx
-rw-r--r--@  1 wmitsuda  staff    43M Jul 25 09:15 v1-004000-004100-headers.seg
-rw-r--r--@  1 wmitsuda  staff   828B Jul 20 14:56 v1-004000-004100-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    20M Jul 21 01:09 v1-004000-004100-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    50M Jul 21 01:08 v1-004000-004100-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    11G Jul 25 09:15 v1-004000-004100-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   107K Jul 20 15:00 v1-004000-004100-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-004100-004200-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-004100-004200-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-004100-004200-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   545K Jul 21 01:04 v1-004100-004200-headers.idx
-rw-r--r--@  1 wmitsuda  staff    43M Jul 25 09:15 v1-004100-004200-headers.seg
-rw-r--r--@  1 wmitsuda  staff   828B Jul 20 14:56 v1-004100-004200-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    25M Jul 21 01:13 v1-004100-004200-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    61M Jul 21 01:12 v1-004100-004200-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   9.5G Jul 25 09:15 v1-004100-004200-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    95K Jul 20 15:00 v1-004100-004200-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-004200-004300-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-004200-004300-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004200-004300-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   544K Jul 21 01:04 v1-004200-004300-headers.idx
-rw-r--r--@  1 wmitsuda  staff    42M Jul 25 09:15 v1-004200-004300-headers.seg
-rw-r--r--@  1 wmitsuda  staff   828B Jul 20 14:56 v1-004200-004300-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    26M Jul 21 01:13 v1-004200-004300-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    62M Jul 21 01:12 v1-004200-004300-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   7.4G Jul 25 09:15 v1-004200-004300-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    75K Jul 20 15:00 v1-004200-004300-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   424K Jul 21 01:04 v1-004300-004400-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-004300-004400-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-004300-004400-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   545K Jul 21 01:04 v1-004300-004400-headers.idx
-rw-r--r--@  1 wmitsuda  staff    44M Jul 25 09:15 v1-004300-004400-headers.seg
-rw-r--r--@  1 wmitsuda  staff   828B Jul 20 14:56 v1-004300-004400-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    23M Jul 21 01:16 v1-004300-004400-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    55M Jul 21 01:16 v1-004300-004400-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   9.9G Jul 25 09:15 v1-004300-004400-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    99K Jul 20 15:00 v1-004300-004400-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-004400-004500-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-004400-004500-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004400-004500-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   545K Jul 21 01:04 v1-004400-004500-headers.idx
-rw-r--r--@  1 wmitsuda  staff    44M Jul 25 09:15 v1-004400-004500-headers.seg
-rw-r--r--@  1 wmitsuda  staff   848B Jul 20 14:56 v1-004400-004500-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    20M Jul 21 01:16 v1-004400-004500-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 21 01:15 v1-004400-004500-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   7.7G Jul 25 09:15 v1-004400-004500-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    77K Jul 20 15:00 v1-004400-004500-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-004500-004600-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-004500-004600-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004500-004600-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   545K Jul 21 01:04 v1-004500-004600-headers.idx
-rw-r--r--@  1 wmitsuda  staff    44M Jul 25 09:15 v1-004500-004600-headers.seg
-rw-r--r--@  1 wmitsuda  staff   848B Jul 20 14:56 v1-004500-004600-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    22M Jul 21 01:18 v1-004500-004600-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    52M Jul 21 01:18 v1-004500-004600-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   5.4G Jul 25 09:15 v1-004500-004600-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    55K Jul 20 15:00 v1-004500-004600-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-004600-004700-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-004600-004700-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004600-004700-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   546K Jul 21 01:04 v1-004600-004700-headers.idx
-rw-r--r--@  1 wmitsuda  staff    45M Jul 25 09:15 v1-004600-004700-headers.seg
-rw-r--r--@  1 wmitsuda  staff   848B Jul 20 14:56 v1-004600-004700-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    21M Jul 21 01:19 v1-004600-004700-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    51M Jul 21 01:18 v1-004600-004700-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   5.5G Jul 25 09:15 v1-004600-004700-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    56K Jul 20 15:00 v1-004600-004700-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-004700-004800-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-004700-004800-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004700-004800-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   546K Jul 21 01:04 v1-004700-004800-headers.idx
-rw-r--r--@  1 wmitsuda  staff    45M Jul 25 09:15 v1-004700-004800-headers.seg
-rw-r--r--@  1 wmitsuda  staff   848B Jul 20 14:56 v1-004700-004800-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    26M Jul 21 01:21 v1-004700-004800-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    62M Jul 21 01:20 v1-004700-004800-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   6.1G Jul 25 09:15 v1-004700-004800-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    61K Jul 20 15:00 v1-004700-004800-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-004800-004900-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-004800-004900-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-004800-004900-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   546K Jul 21 01:04 v1-004800-004900-headers.idx
-rw-r--r--@  1 wmitsuda  staff    46M Jul 25 09:15 v1-004800-004900-headers.seg
-rw-r--r--@  1 wmitsuda  staff   848B Jul 20 14:56 v1-004800-004900-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    26M Jul 21 01:22 v1-004800-004900-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    62M Jul 21 01:21 v1-004800-004900-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   8.0G Jul 25 09:15 v1-004800-004900-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    80K Jul 20 15:00 v1-004800-004900-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   428K Jul 21 01:04 v1-004900-005000-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    14M Jul 25 09:15 v1-004900-005000-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   547B Jul 20 14:56 v1-004900-005000-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   546K Jul 21 01:04 v1-004900-005000-headers.idx
-rw-r--r--@  1 wmitsuda  staff    46M Jul 25 09:15 v1-004900-005000-headers.seg
-rw-r--r--@  1 wmitsuda  staff   868B Jul 20 14:56 v1-004900-005000-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    34M Jul 21 01:23 v1-004900-005000-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    81M Jul 21 01:22 v1-004900-005000-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   8.3G Jul 25 09:15 v1-004900-005000-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    84K Jul 20 15:00 v1-004900-005000-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-005000-005100-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-005000-005100-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005000-005100-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   546K Jul 21 01:04 v1-005000-005100-headers.idx
-rw-r--r--@  1 wmitsuda  staff    46M Jul 25 09:15 v1-005000-005100-headers.seg
-rw-r--r--@  1 wmitsuda  staff   868B Jul 20 14:56 v1-005000-005100-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    35M Jul 21 01:26 v1-005000-005100-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    84M Jul 21 01:25 v1-005000-005100-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    10G Jul 25 09:15 v1-005000-005100-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   105K Jul 20 15:00 v1-005000-005100-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-005100-005200-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005100-005200-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005100-005200-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   547K Jul 21 01:04 v1-005100-005200-headers.idx
-rw-r--r--@  1 wmitsuda  staff    47M Jul 25 09:15 v1-005100-005200-headers.seg
-rw-r--r--@  1 wmitsuda  staff   868B Jul 20 14:56 v1-005100-005200-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    34M Jul 21 01:26 v1-005100-005200-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    80M Jul 21 01:25 v1-005100-005200-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   8.3G Jul 25 09:15 v1-005100-005200-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    83K Jul 20 15:00 v1-005100-005200-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-005200-005300-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005200-005300-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005200-005300-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005200-005300-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005200-005300-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005200-005300-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    38M Jul 21 01:27 v1-005200-005300-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    92M Jul 21 01:27 v1-005200-005300-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   9.5G Jul 25 09:15 v1-005200-005300-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    95K Jul 20 15:00 v1-005200-005300-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-005300-005400-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005300-005400-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-005300-005400-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005300-005400-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005300-005400-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005300-005400-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    35M Jul 21 01:30 v1-005300-005400-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    85M Jul 21 01:29 v1-005300-005400-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    13G Jul 25 09:15 v1-005300-005400-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   132K Jul 20 15:00 v1-005300-005400-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-005400-005500-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005400-005500-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005400-005500-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005400-005500-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005400-005500-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005400-005500-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    30M Jul 21 01:30 v1-005400-005500-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    72M Jul 21 01:29 v1-005400-005500-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   8.0G Jul 25 09:15 v1-005400-005500-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    80K Jul 20 15:00 v1-005400-005500-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-005500-005600-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005500-005600-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005500-005600-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005500-005600-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005500-005600-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005500-005600-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    32M Jul 21 01:31 v1-005500-005600-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    76M Jul 21 01:30 v1-005500-005600-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   6.7G Jul 25 09:15 v1-005500-005600-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    67K Jul 20 15:00 v1-005500-005600-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   424K Jul 21 01:04 v1-005600-005700-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005600-005700-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-005600-005700-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005600-005700-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005600-005700-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005600-005700-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    28M Jul 21 01:33 v1-005600-005700-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    68M Jul 21 01:32 v1-005600-005700-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   7.7G Jul 25 09:15 v1-005600-005700-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    77K Jul 20 15:00 v1-005600-005700-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   424K Jul 21 01:04 v1-005700-005800-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005700-005800-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-005700-005800-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005700-005800-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005700-005800-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005700-005800-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    29M Jul 21 01:34 v1-005700-005800-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    70M Jul 21 01:33 v1-005700-005800-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   9.0G Jul 25 09:15 v1-005700-005800-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    90K Jul 20 15:00 v1-005700-005800-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-005800-005900-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-005800-005900-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005800-005900-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005800-005900-headers.idx
-rw-r--r--@  1 wmitsuda  staff    49M Jul 25 09:15 v1-005800-005900-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005800-005900-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    32M Jul 21 01:35 v1-005800-005900-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    78M Jul 21 01:34 v1-005800-005900-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   9.5G Jul 25 09:15 v1-005800-005900-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    95K Jul 20 15:00 v1-005800-005900-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-005900-006000-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-005900-006000-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-005900-006000-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-005900-006000-headers.idx
-rw-r--r--@  1 wmitsuda  staff    50M Jul 25 09:15 v1-005900-006000-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-005900-006000-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    32M Jul 21 01:37 v1-005900-006000-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    77M Jul 21 01:37 v1-005900-006000-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    13G Jul 25 09:15 v1-005900-006000-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   128K Jul 20 15:00 v1-005900-006000-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   425K Jul 21 01:04 v1-006000-006100-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    12M Jul 25 09:15 v1-006000-006100-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-006000-006100-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-006000-006100-headers.idx
-rw-r--r--@  1 wmitsuda  staff    50M Jul 25 09:15 v1-006000-006100-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-006000-006100-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    36M Jul 21 01:38 v1-006000-006100-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    87M Jul 21 01:37 v1-006000-006100-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    11G Jul 25 09:15 v1-006000-006100-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   111K Jul 20 15:00 v1-006000-006100-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   423K Jul 21 01:04 v1-006100-006200-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    11M Jul 25 09:15 v1-006100-006200-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   507B Jul 20 14:56 v1-006100-006200-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-006100-006200-headers.idx
-rw-r--r--@  1 wmitsuda  staff    50M Jul 25 09:15 v1-006100-006200-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-006100-006200-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    30M Jul 21 01:39 v1-006100-006200-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    72M Jul 21 01:38 v1-006100-006200-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   9.5G Jul 25 09:15 v1-006100-006200-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    95K Jul 20 15:00 v1-006100-006200-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   426K Jul 21 01:04 v1-006200-006300-bodies.idx
-rw-r--r--@  1 wmitsuda  staff    13M Jul 25 09:15 v1-006200-006300-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   527B Jul 20 14:56 v1-006200-006300-bodies.seg.torrent
-rw-r--r--@  1 wmitsuda  staff   548K Jul 21 01:04 v1-006200-006300-headers.idx
-rw-r--r--@  1 wmitsuda  staff    50M Jul 25 09:15 v1-006200-006300-headers.seg
-rw-r--r--@  1 wmitsuda  staff   888B Jul 20 14:56 v1-006200-006300-headers.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    31M Jul 21 01:42 v1-006200-006300-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff    74M Jul 21 01:41 v1-006200-006300-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   8.4G Jul 25 09:15 v1-006200-006300-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    85K Jul 20 15:00 v1-006200-006300-transactions.seg.torrent
-rw-r--r--@  1 wmitsuda  staff    42K Jul 23 11:33 v1-006300-006310-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 23 11:33 v1-006300-006310-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 23 11:33 v1-006300-006310-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 23 11:33 v1-006300-006310-headers.seg
-rw-r--r--@  1 wmitsuda  staff   2.9M Jul 23 11:36 v1-006300-006310-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   6.0M Jul 23 11:36 v1-006300-006310-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   797M Jul 23 11:35 v1-006300-006310-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    42K Jul 23 13:01 v1-006310-006320-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 23 13:01 v1-006310-006320-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 23 13:01 v1-006310-006320-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 23 13:01 v1-006310-006320-headers.seg
-rw-r--r--@  1 wmitsuda  staff   3.0M Jul 23 13:04 v1-006310-006320-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   6.3M Jul 23 13:04 v1-006310-006320-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   917M Jul 23 13:04 v1-006310-006320-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    43K Jul 23 14:30 v1-006320-006330-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 23 14:30 v1-006320-006330-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 23 14:30 v1-006320-006330-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 23 14:30 v1-006320-006330-headers.seg
-rw-r--r--@  1 wmitsuda  staff   3.2M Jul 23 14:33 v1-006320-006330-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   6.7M Jul 23 14:33 v1-006320-006330-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   930M Jul 23 14:32 v1-006320-006330-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    43K Jul 23 16:18 v1-006330-006340-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 23 16:18 v1-006330-006340-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 23 16:18 v1-006330-006340-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 23 16:18 v1-006330-006340-headers.seg
-rw-r--r--@  1 wmitsuda  staff   3.2M Jul 23 16:29 v1-006330-006340-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   6.8M Jul 23 16:28 v1-006330-006340-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   883M Jul 23 16:28 v1-006330-006340-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    43K Jul 24 02:11 v1-006340-006350-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 24 02:11 v1-006340-006350-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 24 02:11 v1-006340-006350-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 24 02:11 v1-006340-006350-headers.seg
-rw-r--r--@  1 wmitsuda  staff   3.0M Jul 24 02:13 v1-006340-006350-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   6.3M Jul 24 02:13 v1-006340-006350-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   831M Jul 24 02:13 v1-006340-006350-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    43K Jul 24 11:54 v1-006350-006360-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 24 11:54 v1-006350-006360-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 24 11:54 v1-006350-006360-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 24 11:54 v1-006350-006360-headers.seg
-rw-r--r--@  1 wmitsuda  staff   3.1M Jul 24 12:04 v1-006350-006360-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   6.5M Jul 24 12:04 v1-006350-006360-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   858M Jul 24 12:03 v1-006350-006360-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   375K Jul 24 12:05 v1-006360-006361-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   779K Jul 24 12:05 v1-006360-006361-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    98M Jul 24 12:05 v1-006360-006361-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    43K Jul 25 15:09 v1-006360-006370-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 25 15:09 v1-006360-006370-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 25 15:09 v1-006360-006370-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 25 15:09 v1-006360-006370-headers.seg
-rw-r--r--@  1 wmitsuda  staff   346K Jul 24 12:06 v1-006361-006362-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   723K Jul 24 12:06 v1-006361-006362-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   105M Jul 24 12:06 v1-006361-006362-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   339K Jul 24 12:07 v1-006362-006363-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   708K Jul 24 12:07 v1-006362-006363-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   101M Jul 24 12:07 v1-006362-006363-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   389K Jul 24 12:08 v1-006363-006364-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   807K Jul 24 12:08 v1-006363-006364-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    87M Jul 24 12:08 v1-006363-006364-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   305K Jul 24 12:09 v1-006364-006365-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   634K Jul 24 12:09 v1-006364-006365-transactions.idx
-rw-r--r--@  1 wmitsuda  staff    76M Jul 24 12:09 v1-006364-006365-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   316K Jul 25 09:17 v1-006365-006366-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   661K Jul 25 09:17 v1-006365-006366-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   104M Jul 25 09:17 v1-006365-006366-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   375K Jul 25 15:04 v1-006366-006367-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   784K Jul 25 15:04 v1-006366-006367-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   111M Jul 25 15:04 v1-006366-006367-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   350K Jul 25 15:06 v1-006367-006368-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   731K Jul 25 15:06 v1-006367-006368-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   105M Jul 25 15:06 v1-006367-006368-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   364K Jul 25 15:08 v1-006368-006369-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   762K Jul 25 15:07 v1-006368-006369-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   113M Jul 25 15:07 v1-006368-006369-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   308K Jul 25 15:09 v1-006369-006370-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   648K Jul 25 15:09 v1-006369-006370-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   117M Jul 25 15:09 v1-006369-006370-transactions.seg
-rw-r--r--@  1 wmitsuda  staff    43K Jul 27 05:11 v1-006370-006380-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   1.1M Jul 27 05:11 v1-006370-006380-bodies.seg
-rw-r--r--@  1 wmitsuda  staff    45K Jul 27 05:11 v1-006370-006380-headers.idx
-rw-r--r--@  1 wmitsuda  staff   5.0M Jul 27 05:11 v1-006370-006380-headers.seg
-rw-r--r--@  1 wmitsuda  staff   3.4M Jul 27 05:25 v1-006370-006380-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   7.1M Jul 27 05:25 v1-006370-006380-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   1.2G Jul 27 05:24 v1-006370-006380-transactions.seg
-rw-r--r--@  1 wmitsuda  staff   4.9K Jul 27 05:25 v1-006380-006381-bodies.idx
-rw-r--r--@  1 wmitsuda  staff   538K Jul 27 05:25 v1-006380-006381-bodies.seg
-rw-r--r--@  1 wmitsuda  staff   4.9K Jul 27 05:25 v1-006380-006381-headers.idx
-rw-r--r--@  1 wmitsuda  staff   597K Jul 27 05:25 v1-006380-006381-headers.seg
-rw-r--r--@  1 wmitsuda  staff   334K Jul 27 05:25 v1-006380-006381-transactions-to-block.idx
-rw-r--r--@  1 wmitsuda  staff   710K Jul 27 05:25 v1-006380-006381-transactions.idx
-rw-r--r--@  1 wmitsuda  staff   186M Jul 27 05:25 v1-006380-006381-transactions.seg

@wmitsuda show

go run ./cmd/integration print_stages  --log.console.verbosity=5

and after this - you can fix node by rm -rf chaindata

go run ./cmd/integration print_stages --log.console.verbosity=5

go run ./cmd/integration print_stages  --log.console.verbosity=5 --datadir=/Volumes/erigon3-sepolia/erigon-backup-issue-with-gap-bug --chain=sepolia
INFO[08-01|14:29:02.635] logging to file system                   log dir=/Volumes/erigon3-sepolia/erigon-backup-issue-with-gap-bug/logs file prefix=integration log level=info json=false
INFO[08-01|14:29:02.641] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
INFO[08-01|14:29:02.648] [downloads]                              locked=true at="21 Jul 24 01:04 2024"
INFO[08-01|14:29:02.682] [snapshots:caplin] Stat                  blocks=0k indices=0k
DBUG[08-01|14:29:02.683] [snapshots] see gap                      type=transactions from=6360000
DBUG[08-01|14:29:02.683] [snapshots] see gap                      type=transactions from=6360000
DBUG[08-01|14:29:17.688] WarmUp finished                          file=v1-code.0-128.kv M=256 N=15778334 cached="61634 %0.39062" cacheSize="3.5 MB" fileSize="16.2 GB"
DBUG[08-01|14:29:18.518] WarmUp finished                          file=v1-accounts.0-128.kv M=256 N=25763901 cached="100640 %0.39062" cacheSize="5.8 MB" fileSize="1.2 GB"
DBUG[08-01|14:29:19.735] WarmUp finished                          file=v1-code.128-160.kv M=256 N=2008538 cached="7845 %0.39058" cacheSize="459.7 KB" fileSize="4.1 GB"
DBUG[08-01|14:29:20.995] WarmUp finished                          file=v1-code.160-176.kv M=256 N=1277317 cached="4989 %0.39058" cacheSize="292.3 KB" fileSize="2.0 GB"
DBUG[08-01|14:29:21.073] WarmUp finished                          file=v1-code.176-177.kv M=256 N=66009 cached="257 %0.38934" cacheSize="15.1 KB" fileSize="150.1 MB"
DBUG[08-01|14:29:22.043] WarmUp finished                          file=v1-accounts.128-160.kv M=256 N=5448181 cached="21281 %0.39061" cacheSize="1.2 MB" fileSize="226.6 MB"
DBUG[08-01|14:29:24.287] WarmUp finished                          file=v1-accounts.160-176.kv M=256 N=3692178 cached="14422 %0.39061" cacheSize="845.0 KB" fileSize="152.2 MB"
DBUG[08-01|14:29:24.567] WarmUp finished                          file=v1-accounts.176-177.kv M=256 N=400083 cached="1562 %0.39042" cacheSize="91.5 KB" fileSize="14.9 MB"
DBUG[08-01|14:30:27.019] WarmUp finished                          file=v1-commitment.0-128.kv M=256 N=94133941 cached="367710 %0.39062" cacheSize="25.2 MB" fileSize="19.0 GB"
DBUG[08-01|14:30:46.925] WarmUp finished                          file=v1-commitment.128-160.kv M=256 N=22592219 cached="88250 %0.39062" cacheSize="5.7 MB" fileSize="3.3 GB"
DBUG[08-01|14:31:02.786] WarmUp finished                          file=v1-commitment.160-176.kv M=256 N=16791758 cached="65592 %0.39062" cacheSize="4.2 MB" fileSize="2.9 GB"
DBUG[08-01|14:31:05.640] WarmUp finished                          file=v1-commitment.176-177.kv M=256 N=2878817 cached="11245 %0.39061" cacheSize="726.5 KB" fileSize="876.7 MB"
DBUG[08-01|14:31:22.319] WarmUp finished                          file=v1-storage.0-128.kv M=256 N=242640549 cached="947814 %0.39062" cacheSize="83.2 MB" fileSize="15.1 GB"
DBUG[08-01|14:31:42.602] WarmUp finished                          file=v1-storage.128-160.kv M=256 N=38237329 cached="149364 %0.39062" cacheSize="13.1 MB" fileSize="2.5 GB"
DBUG[08-01|14:31:55.026] WarmUp finished                          file=v1-storage.160-176.kv M=256 N=27346198 cached="106821 %0.39062" cacheSize="9.4 MB" fileSize="1.7 GB"
DBUG[08-01|14:31:56.205] WarmUp finished                          file=v1-storage.176-177.kv M=256 N=1940874 cached="7581 %0.39060" cacheSize="681.1 KB" fileSize="122.0 MB"
INFO[08-01|14:31:56.220] [snapshots:blocks] Stat                  blocks=6360k indices=6360k alloc=1.1GB sys=1.2GB
INFO[08-01|14:31:56.221] [snapshots:bor] Stat                     blocks=0k indices=0k alloc=1.1GB sys=1.2GB
INFO[08-01|14:31:56.245] [snapshots:history] Stat                 blocks=6373k txs=276m txNum2blockNum="128=5623K,160=6107K,176=6359K,177=6373K" first_history_idx_in_db=6373457 last_comitment_block=6373457 last_comitment_tx_num=276562500 alloc=1.1GB sys=1.2GB
Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.

                                 stage_at        prune_at
Snapshots                        6384435         0
Headers                          6384435         0
BorHeimdall                      0               0
PolygonSync                      0               0
BlockHashes                      6384435         0
Bodies                           6384435         0
Senders                          6384435         0
Execution                        6384435         6384435
CustomTrace                      0               0
Translation                      0               0
HashState                        0               0
IntermediateHashes               0               0
AccountHistoryIndex              0               0
StorageHistoryIndex              0               0
LogIndex                         0               0
CallTraces                       0               0
TxLookup                         6384435         6306370
Finish                           6384435         0
--
prune distance:

blocks.v2: true, segments=6359999, indices=6359999
blocks.bor.v2: segments=0, indices=0

state.history: idx steps: 0.72, lastBlockInSnap=6373457, TxNums_Index(6384435,277694336), filesAmount: [4 4 4 4 5 5 5 5]

sequence: EthTx=277694337

in db: first header 6380000, last header 6384435, first body 6380000, last body 6384435
--

and after this - you can fix node by rm -rf chaindata

thanks, I already restored it from a healthy backup, but I kept the one with the bug as a backup if you need to run something on it.

just for the record, it happened today again with an updated main (commit b85ab1833e2e83328110d01d34f11b92fc477383):

INFO[08-04|13:39:48.186] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_2/ssz_snappy
INFO[08-04|13:39:48.186] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/bls_to_execution_change/ssz_snappy
INFO[08-04|13:39:48.186] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_4/ssz_snappy
INFO[08-04|13:39:48.188] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/beacon_block/ssz_snappy
INFO[08-04|13:39:48.188] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/proposer_slashing/ssz_snappy
WARN[08-04|13:39:48.325] [1/6 OtterSync] Some blocks are not in snapshots and not in db. this could have happend due to the node being stopped at the wrong time, you can fix this with 'rm -rf /Volumes/erigon3-sepolia/erigon/chaindata' (this is not equivalent to a full resync) max_in_snapshots=6299999 min_in_db=6408000
EROR[08-04|13:39:48.325] Could not start execution service        err="[1/6 OtterSync] some blocks are not in snapshots and not in db. this could have happend due to the node being stopped at the wrong time, you can fix this with 'rm -rf /Volumes/erigon3-sepolia/erigon/chaindata' (this is not equivalent to a full resync)"
INFO[08-04|13:40:06.039] Missing blocks download finished (note: this does not mean that the history is complete, only that the missing blocks need for sync have been downloaded) app=caplin stage=DownloadHistoricalBlocks
INFO[08-04|13:40:07.227] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
INFO[08-04|13:40:07.235] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=5587904 to=5588000
INFO[08-04|13:40:38.571] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=5587967 distance-from-chain-tip=6m36s estimated-time-remaining=15s
INFO[08-04|13:40:47.187] P2P                                      app=caplin peers=13
INFO[08-04|13:41:09.276] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=5588005 distance-from-chain-tip=-1m0s estimated-time-remaining=0s
WARN[08-04|13:41:09.462] failed to insert blocks                  err="parent's total difficulty not found with hash 3637dad49f9c7d037e68a045ed42dd182994487446ec1994316ddd9328a39798 and height 6436374: <nil>"
WARN[08-04|13:41:09.535] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of new fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: forkchoice: block d4064682180c840728cd4de047e30d7d55f87658e22c42dc0fade3b455a42ec6 not found or was marked invalid"
INFO[08-04|13:41:47.187] P2P                                      app=caplin peers=14
WARN[08-04|13:42:44.023] [p2p] No GoodPeers
INFO[08-04|13:42:44.023] [mem] memory stats                       alloc=1.9GB sys=3.4GB
INFO[08-04|13:42:45.040] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.3GB sys=3.4GB
INFO[08-04|13:42:47.187] P2P                                      app=caplin peers=13
INFO[08-04|13:43:47.186] P2P                                      app=caplin peers=8

I got this one again today and I think it is being quite problematic as I think the "remove chaindata" workaround does not work reliably anymore, as the next restart it gets stuck by "too far unwind" and the only solution seems to be restore from a backup.

logs from the "missing blocks" corruption:

INFO[08-11|15:48:46.731] [1/6 OtterSync] Requesting downloads
WARN[08-11|15:48:46.867] [1/6 OtterSync] Some blocks are not in snapshots and not in db. this could have happend due to the node being stopped
 at the wrong time, you can fix this with 'rm -rf /Users/wmitsuda/eth-nodes/erigon3-sepolia/chaindata' (this is not equivalent to a full resyn
c) max_in_snapshots=6449999 min_in_db=6465100
EROR[08-11|15:48:46.867] Could not start execution service        err="[1/6 OtterSync] some blocks are not in snapshots and not in db. this co
uld have happend due to the node being stopped at the wrong time, you can fix this with 'rm -rf /Users/wmitsuda/eth-nodes/erigon3-sepolia/chai
ndata' (this is not equivalent to a full resync)"
meta_unsteady:10335 wipe txn #3, meta 2
INFO[08-11|15:48:47.373] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/blob_sidecar_5/ssz_snappy
INFO[08-11|15:48:47.373] [Gossip] Subscribed to topic             topic=/eth2/d31f6191/proposer_slashing/ssz_snappy

then I deleted chainddata and on the next run:

INFO[08-11|17:02:03.315] [Caplin] Inserted blocks                 progress=6479999
WARN[08-11|17:02:03.416] failed to update fork choice             err="execution Client RPC failed to retrieve ForkChoiceUpdate response, err:
 too far unwind. requested=6449999, minAllowed=6472448"
INFO[08-11|17:02:03.925] [Caplin] Inserting blocks                from=6480000 to=6480999
INFO[08-11|17:02:04.242] [Caplin] Inserted blocks                 progress=6480999
INFO[08-11|17:02:26.711] P2P                                      app=caplin peers=4
INFO[08-11|17:03:26.711] P2P                                      app=caplin peers=11
WARN[08-11|17:04:22.130] [engine] CurrentHeader                   err="ethereumExecutionModule.CurrentHeader: could not begin database tx cont
ext deadline exceeded"
INFO[08-11|17:04:26.435] [p2p] GoodPeers                          eth67=2
INFO[08-11|17:04:26.436] [mem] memory stats                       alloc=1.6GB sys=4.7GB
INFO[08-11|17:04:26.438] [txpool] stat                            pending=0 baseFee=0 queued=5423 alloc=1.6GB sys=4.7GB
INFO[08-11|17:04:26.710] P2P                                      app=caplin peers=5
INFO[08-11|17:05:26.710] P2P                                      app=caplin peers=3
WARN[08-11|17:05:54.690] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: too far unwind. requested=644
9999, minAllowed=6472448"
WARN[08-11|17:06:09.606] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: too far unwind. requested=644
9999, minAllowed=6472448"
WARN[08-11|17:06:20.331] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: too far unwind. requested=644
9999, minAllowed=6472448"
INFO[08-11|17:06:26.709] P2P                                      app=caplin peers=9
INFO[08-11|17:07:26.433] [p2p] GoodPeers                          eth67=2
INFO[08-11|17:07:26.435] [mem] memory stats                       alloc=2.5GB sys=4.7GB
INFO[08-11|17:07:26.437] [txpool] stat                            pending=0 baseFee=0 queued=5466 alloc=2.5GB sys=4.7GB
INFO[08-11|17:07:26.709] P2P                                      app=caplin peers=17
INFO[08-11|17:08:26.708] P2P                                      app=caplin peers=17
INFO[08-11|17:09:26.708] P2P                                      app=caplin peers=9
INFO[08-11|17:10:26.432] [mem] memory stats                       alloc=2.8GB sys=4.7GB
INFO[08-11|17:10:26.432] [p2p] GoodPeers                          eth67=2
INFO[08-11|17:10:26.435] [txpool] stat                            pending=0 baseFee=0 queued=5492 alloc=2.8GB sys=4.7GB
INFO[08-11|17:10:26.707] P2P                                      app=caplin peers=6
INFO[08-11|17:11:26.707] P2P                                      app=caplin peers=5
INFO[08-11|17:12:26.706] P2P                                      app=caplin peers=15

BTW, this situation is exactly what I described here: https://github.com/erigontech/erigon/issues/11494

so, this issue leads to that another one.

found a workaround, but that involves:

restore from a healthy backup
disable caplin
run lighthouse against erigon in order to make it advance the chain to the tip
reenable caplin once on chain tip

I’ll try it as soon as it get merged, following it

I reproduced 1 case when "transaction.seg" was manually deleted - got same error. Fixed by https://github.com/erigontech/erigon/pull/11557

erigontech / erigon

[e3] Node apparently got corrupted #11381