search

bnb-chain / bsc-snapshots

252 stars 89 forks source link

The hard disk space is exhausted when indexing snapshot. #387

Closed bigjoker007 closed 2 weeks ago

bigjoker007 commented 1 month ago

I have started a new Erigon node, using version 1.2.15 and the snapshot from 20240803. When everything is new, I had 12TB of available disk space. Then, I did the following:

  1. Created the chaindata and parlia folders inside the /ssd/data/erigon_datadir directory.
  2. Move the mdbx from snapshot to /ssd/data/erigon_datadir/chaindata.
  3. Download the parlia mdbx to /ssd/data/erigon_datadir/parlia.
  4. Run the erigon.
  5. After completing the download of the 1.1TB snapshot, there has 3.2TB of available disk space. The log of download like this: 
    [INFO] [09-12|11:07:49.504] [1/15 Snapshots] download                progress="99.55% 1.1TB/1.1TB" time-left=999hrs:99m total-time=25h21m0s download=0B/s upload=0B/s peers=0 files=330 metadata=330/330 connections=0 alloc=1.1GB sys=4.1GB
[INFO] [09-12|11:08:08.567] [snapshots] no progress yet              files=8 list=v1-040900-041000-transactions.seg,v1-040900-041000-headers.seg,v1-040900-041000-bodies.seg,v1-040800-040900-bodies.seg,v1-040800-040900-headers.seg,...
  6. After running index snapshot for about 15 hours, there was only 180GB of available disk space left, and still decreasing. The log of index like this: 
    [INFO] [09-13|10:56:22.105] [1/15 Snapshots] Indexing                progress="v1-007000-007500-transactions.seg=1%, v1-009500-010000-transactions.seg=2%, v1-010000-010500-transactions.seg=3%, v1-010500-011000-transactions.seg=3%, v1-011000-011500-transactions.seg=3%, v1-011500-012000-transactions.seg=2%, v1-012000-012500-transactions.seg=2%" total-indexing-time=1m0s alloc=5.3GB sys=12.2GB
[INFO] [09-13|10:56:42.107] [1/15 Snapshots] Indexing                progress="v1-007000-007500-transactions.seg=2%, v1-009500-010000-transactions.seg=3%, v1-010000-010500-transactions.seg=4%, v1-010500-011000-transactions.seg=4%, v1-011000-011500-transactions.seg=4%, v1-011500-012000-transactions.seg=3%, v1-012000-012500-transactions.seg=2%" total-indexing-time=1m20s alloc=5.5GB sys=12.2GB
[INFO] [09-13|10:57:02.106] [1/15 Snapshots] Indexing                progress="v1-007000-007500-transactions.seg=3%, v1-009500-010000-transactions.seg=4%, v1-010000-010500-transactions.seg=5%, v1-010500-011000-transactions.seg=5%, v1-011000-011500-transactions.seg=6%, v1-011500-012000-transactions.seg=4%, v1-012000-012500-transactions.seg=3%" total-indexing-time=1m40s alloc=5.7GB sys=12.2GB
[INFO] [09-13|10:57:22.106] [1/15 Snapshots] Indexing                progress="v1-007000-007500-transactions.seg=3%, v1-009500-010000-transactions.seg=4%, v1-010000-010500-transactions.seg=6%, v1-010500-011000-transactions.seg=6%, v1-011000-011500-transactions.seg=7%, v1-011500-012000-transactions.seg=5%, v1-012000-012500-transactions.seg=4%" total-indexing-time=2m0s alloc=5.9GB sys=12.2GB
[INFO] [09-13|10:57:42.105] [1/15 Snapshots] Indexing                progress="v1-007000-007500-transactions.seg=4%, v1-009500-010000-transactions.seg=5%, v1-010000-010500-transactions.seg=7%, v1-010500-011000-transactions.seg=8%, v1-011000-011500-transactions.seg=8%, v1-011500-012000-transactions.seg=6%, v1-012000-012500-transactions.seg=5%" total-indexing-time=2m20s alloc=4.5GB sys=12.2GB
[INFO] [09-13|10:58:02.106] [1/15 Snapshots] Indexing                progress="v1-007000-007500-transactions.seg=5%, v1-009500-010000-transactions.seg=6%, v1-010000-010500-transactions.seg=8%, v1-010500-011000-transactions.seg=9%, v1-011000-011500-transactions.seg=9%, v1-011500-012000-transactions.seg=7%, v1-012000-012500-transactions.seg=5%" total-indexing-time=2m40s alloc=4.7GB sys=12.2GB
  7. There is no disk space left now. The log is: 
    [INFO] [09-13|14:16:18.740] [1/15 Snapshots] Requesting downloads 
[INFO] [09-13|14:16:21.379] [snapshots:download] Stat                blocks=41000k indices=7000k alloc=3.5GB sys=13.4GB
[INFO] [09-13|14:16:21.380] [snapshots:missed-idx] Stat              blocks=41000k indices=7000k alloc=3.5GB sys=13.4GB
[EROR] [09-13|14:16:24.818] Staged Sync                              err="[1/15 Snapshots] can't build missed indices: v1-007000-007500-transactions.seg: open /ssd/data/erigon/temp/erigon-lfp-buf-860367166: no space left on device"
STdevK commented 1 month ago

what's the node start-up command?

bigjoker007 commented 1 month ago

what's the node start-up command?

./erigon --log.dir.path /hdd/app/BscNode/logs --datadir /ssd/data/erigon_datadir --chain=bsc --http --http.port=6666 --http.addr 0.0.0.0 --http.api=web3,net,eth,debug,trace,txpool --db.pagesize=16k --p2p.protocol=68 --metrics --txpool.disable

bigjoker007 commented 1 month ago

I tried deleting all the data under the /ssd/data/erigon_datadir and started the node, then it need to download of the 1.1TB snapshot, and the log output was the same as in step 5. It seems that the Erigon node failed to properly recognize /ssd/data/erigon_datadir/chaindata/mdbx.dat. This appears to be the same issue as https://github.com/bnb-chain/bsc-snapshots/issues/378#issue-2435385453.

du5 commented 1 month ago

download this https://github.com/48Club/bsc-snapshots?tab=readme-ov-file#erigon-snapshot-directory

https://snapshots.48.club/snapshots.42000000.tar.zst