search

48Club / bsc-snapshots

bsc daily snapshot
https://www.48.club
157 stars 21 forks source link

Erigon Snapshot Help, Q&A #82

Open du5 opened 1 year ago

du5 commented 1 year ago

Data verification. 数据校验

It's advisable to verify the SHA256 checksum of the files before deploying them in a production environment. When using a snapshot for the first time or after re-downloading a snapshot, it's recommended to add the --downloader.verify flag during the initial synchronization to ensure data integrity.

建议在投入生产环境之前，校验文件 sha256 特征码，第一次使用快照或者重新下载快照之后建议第一次同步增加 flags --downloader.verify

Synchronization optimization. 同步优化

Regarding synchronization efficiency issues, after extensive testing by our team, we recommend modifying the following flags. They can address the majority of problems encountered during Erigon synchronization.

关于同步效率问题，经过团队的长期测试，我们建议修改以下 flags，它可以解决 Erigon 同步遇到的绝大部分问题

--port=30303 --p2p.protocol=68 --nat=any

File Structure. 文件结构

> du -h ~/.local/share/erigon # complete structure
6.3T    /root/.local/share/erigon/chaindata
25M    /root/.local/share/erigon/parlia
27M     /root/.local/share/erigon/snapshots/db
**748G**    /root/.local/share/erigon/snapshots
7.0T    /root/.local/share/erigon

The above is the directory structure of a typical Erigon node. After extracting the provided snapshot, your directory should look like the following.

以上是一个正常的 Erigon 节点的目录结构，解压完成我们提供的快照后你得到的目录应该是下面这样的 

> du -h ~/.local/share/erigon # torrent missing 
6.3T    /root/.local/share/erigon/chaindata
25M    /root/.local/share/erigon/parlia
27M     /root/.local/share/erigon/snapshots/db
**27M**    /root/.local/share/erigon/snapshots
6.3T    /root/.local/share/erigon

If you encounter the following log during the synchronization process, it means your node is missing the torrent file.

如果你在同步过程中发现以下日志则说明您的节点缺失 torrent 文件

INFO[07-22|02:35:23.798] [1/15 Snapshots] Fetching torrent files metadata 
INFO[07-22|02:35:43.808] [1/15 Snapshots] Waiting for torrents metadata: 41/177 
INFO[07-22|02:36:03.847] [1/15 Snapshots] Waiting for torrents metadata: 100/177 
INFO[07-22|02:36:23.807] [1/15 Snapshots] Waiting for torrents metadata: 156/177 
INFO[07-22|02:36:43.813] [1/15 Snapshots] download                progress="0.03% 246.1MB/705.0GB" download-time-left=22hrs:57m total-download-time=1m20s download=8.7MB/s upload=0B/s

It is evident that the torrent file is missing from /root/.local/share/erigon/snapshots.

We have observed that for both the Erigon fast node and Erigon archive node, the torrent data remains unchanged regardless of the synchronization method.

As a result, we have separately provided all torrent files within the snapshots folder. After downloading the compressed archive, you will need to manually move the extracted files to align with the complete directory structure.

If you find this step too challenging, you can also skip it, as Erigon will automatically fix the missing files for you.

明显可以发现 /root/.local/share/erigon/snapshots 缺失了 torrent 文件

我们发现 Erigon fast node 和 Erigon archive node 他们不会因为同步方式而改变 torrent 数据

因此我们将 snapshots 文件夹内的所有 torrent 文件单独发布，你下载压缩包后需要手动移动解压后的文件，使其和完整结构保持一致

如果此步骤对于你过于困难，你也可以跳过此步骤，Erigon 将自动修复缺失的这部分文件

Update database. 更新数据库

Because we've separated the upload of Torrent files, this approach also saves approximately 1200GB of additional data transfer when updating your node. All you need to do is download the snapshot, overwrite it during extraction, and there's no need to delete the previously downloaded and extracted Torrent files or download the Torrent file archive again. Erigon, when started, will detect the small portion of missing files and automatically repair them. In most cases, this process will be completed quickly.

因为我们拆分开了 Torrent 文件的上传，这样在更新节点时也会节约 1200G 的额外流量，你要做的事就是下载快照覆盖解压，不需要删除上一次下载解压的 Torrent 文件，也不需要重新下载 Torrent 文件压缩包，Erigon 启动后它会检测到缺失的一小部分，它将自动修复这些文件，一般情况下这将很快完成

alexjingjing commented 1 year ago

erigon version 1.1.9-dev 使用了0830的torrent文件进行同步

有报错： [EROR] [10-08|13:07:44.445] Staged Sync err="[1/15 Snapshots] unknown ancestor" 然后就一直循环卡在这步，请问这个该怎么解决？

du5 commented 1 year ago

erigon version 1.1.9-dev 使用了0830的torrent文件进行同步

有报错： [EROR] [10-08|13:07:44.445] Staged Sync err="[1/15 Snapshots] unknown ancestor" 然后就一直循环卡在这步，请问这个该怎么解决？

从创世区块同步还是从快照同步

alexjingjing commented 1 year ago

从快照同步， 最开始下载了https://snapshots.48.club/erigon.fast.31306137.tar.zst这个文件 看到了你在github这个issue下面的回答，https://github.com/node-real/bsc-erigon/issues/169#issuecomment-1662094647

又重新下载了https://snapshots.48.club/erigon.torrent.20230830.tar.zst，解压后放入datadir下面的snapshots文件夹，但是没有按照预期的跳过1/15 现在1/15就不断地在download、verify、Staged Sync err="[1/15 Snapshots] unknown ancestor" 循环

启动参数：erigon --config ./config.yaml --snapshots config.yaml

datadir : '/home/ec2-user/erigon' port : 30311 p2p.protocol: '66,67' nat: 'any' http : true ws: true private.api.addr : "127.0.0.1:9090" chain: 'bsc' db.pagesize: '16k' prune.h.older: 90000 prune.r.older: 90000 prune.t.older: 90000 prune.c.older: 90000 sentry.drop-useless-peers: true downloader.verify: false

maxpeers: 200 http.api : ["eth","debug","net","txpool","web3"] snap.stop: true torrent.download.rate: '100mb'

du5 commented 1 year ago

请提供日志

alexjingjing commented 1 year ago

5分钟前我做了如下操作 1、https://snapshots.48.club/erigon.torrent.20230830.tar.zst 所有文件解压至datadir的snapshots文件夹 2、https://snapshots.48.club/erigon.fast.31306137.tar.zst 解压至datadir的chaindata和parlia文件夹

[INFO] [10-08|15:30:36.731] [1/15 Snapshots] Fetching torrent files metadata [INFO] [10-08|15:30:36.778] [snapshots] Blocks Stat blocks=29500k indices=29500k alloc=3.0GB sys=3.1GB [INFO] [10-08|15:30:36.785] [2/15 Headers] Waiting for headers... from=31306138 [INFO] [10-08|15:30:53.361] [parlia] snapshots build, gather headers block=31300000 [INFO] [10-08|15:30:55.381] [parlia] snapshots build, recover from headers block=31300000 [INFO] [10-08|15:30:55.790] [parlia] snapshots build, gather headers block=31300000 [INFO] [10-08|15:30:58.108] [parlia] snapshots build, recover from headers block=31300000 [INFO] [10-08|15:30:58.493] [2/15 Headers] Inserting headers progress=31306138 queue=1 [INFO] [10-08|15:31:16.785] [2/15 Headers] Inserting headers progress=31322790 queue=1 [INFO] [10-08|15:31:23.491] [txpool] Started [INFO] [10-08|15:31:36.785] [2/15 Headers] Inserting headers progress=31341936 queue=1 [INFO] [10-08|15:31:56.785] [2/15 Headers] Inserting headers progress=31359936 queue=1 [INFO] [10-08|15:32:16.786] [2/15 Headers] Inserting headers progress=31379205 queue=1

从日志输出来看，好像是开始追高度了，不知道我理解的是否正确。

image
du5 commented 1 year ago

5分钟前我做了如下操作 1、https://snapshots.48.club/erigon.torrent.20230830.tar.zst 所有文件解压至datadir的snapshots文件夹 2、https://snapshots.48.club/erigon.fast.31306137.tar.zst 解压至datadir的chaindata和parlia文件夹

[INFO] [10-08|15:30:36.731] [1/15 Snapshots] Fetching torrent files metadata [INFO] [10-08|15:30:36.778] [snapshots] Blocks Stat blocks=29500k indices=29500k alloc=3.0GB sys=3.1GB [INFO] [10-08|15:30:36.785] [2/15 Headers] Waiting for headers... from=31306138 [INFO] [10-08|15:30:53.361] [parlia] snapshots build, gather headers block=31300000 [INFO] [10-08|15:30:55.381] [parlia] snapshots build, recover from headers block=31300000 [INFO] [10-08|15:30:55.790] [parlia] snapshots build, gather headers block=31300000 [INFO] [10-08|15:30:58.108] [parlia] snapshots build, recover from headers block=31300000 [INFO] [10-08|15:30:58.493] [2/15 Headers] Inserting headers progress=31306138 queue=1 [INFO] [10-08|15:31:16.785] [2/15 Headers] Inserting headers progress=31322790 queue=1 [INFO] [10-08|15:31:23.491] [txpool] Started [INFO] [10-08|15:31:36.785] [2/15 Headers] Inserting headers progress=31341936 queue=1 [INFO] [10-08|15:31:56.785] [2/15 Headers] Inserting headers progress=31359936 queue=1 [INFO] [10-08|15:32:16.786] [2/15 Headers] Inserting headers progress=31379205 queue=1

从日志输出来看，好像是开始追高度了，不知道我理解的是否正确。

image

正确操作

alexjingjing commented 1 year ago

嗯嗯，非常感谢。 我感觉是我把散落的一些对于bsc-erigon如何部署+同步的内容给搞混了。

其实就像这个issue描述的一样。 就是把fastnode的chaindata和parlia解压到erigon的datadir就可以开始同步了。 erigon启动后会自动开始下载torrent。只不过需要等待下载700多个G。 如果不想erigon去下载torrent，就用https://snapshots.48.club/erigon.torrent.20230830.tar.zst 这个包，解压缩到datadir的snapshots，erigon可能就只需要下载几个G的补充文件 1/15这一步就过去了。

再次感谢你的解答和时间

alexjingjing commented 1 year ago

请问下7/15 的速度会很慢么，平均2个blocks / sec

[INFO] [10-09|01:31:58.604] [7/15 Execution] Executed blocks number=31359927 blk/s=4.0 tx/s=504.5 Mgas/s=51.0 gasState=0.21 batch=252.7MB alloc=4.9GB sys=10.4GB [INFO] [10-09|01:32:18.565] [7/15 Execution] Executed blocks number=31359978 blk/s=2.6 tx/s=356.1 Mgas/s=40.2 gasState=0.21 batch=254.4MB alloc=5.4GB sys=10.4GB [INFO] [10-09|01:32:38.804] [7/15 Execution] Executed blocks number=31360018 blk/s=2.0 tx/s=259.0 Mgas/s=32.3 gasState=0.21 batch=255.7MB alloc=5.9GB sys=10.4GB

感觉追上高度要一周了

du5 commented 1 year ago

请问下7/15 的速度会很慢么，平均2个blocks / sec

[INFO] [10-09|01:31:58.604] [7/15 Execution] Executed blocks number=31359927 blk/s=4.0 tx/s=504.5 Mgas/s=51.0 gasState=0.21 batch=252.7MB alloc=4.9GB sys=10.4GB [INFO] [10-09|01:32:18.565] [7/15 Execution] Executed blocks number=31359978 blk/s=2.6 tx/s=356.1 Mgas/s=40.2 gasState=0.21 batch=254.4MB alloc=5.4GB sys=10.4GB [INFO] [10-09|01:32:38.804] [7/15 Execution] Executed blocks number=31360018 blk/s=2.0 tx/s=259.0 Mgas/s=32.3 gasState=0.21 batch=255.7MB alloc=5.9GB sys=10.4GB

感觉追上高度要一周了

Mgas/s=40.2 受限于你的磁盘和CPU性能，这个速度很难追上

alexjingjing commented 1 year ago

请问erigon的fastnode 快照包什么时候能再打一个？ 现在的是0830的，要追一个半月的高度真的是有点痛苦。

5/15,6/15动辄都要4h+了

du5 commented 1 year ago

请问erigon的fastnode 快照包什么时候能再打一个？ 现在的是0830的，要追一个半月的高度真的是有点痛苦。

5/15,6/15动辄都要4h+了

正在从创世区块重新同步，可能还需要一周左右

du5 commented 11 months ago

28171ee40074840028d4ac97559476bee7352503

newbie-ksc commented 1 week ago

请教一个场景；

我下载了上一个版本的归档快照，然后开始同步；过了一段时间后，发现咱们有新的快照可以下载了，我想下载最新的归档快照，这样是不是只会追加最近一段时间的区块数据；还是说，又是完整的下载一份快照数据？

du5 commented 1 week ago

请教一个场景；

我下载了上一个版本的归档快照，然后开始同步；过了一段时间后，发现咱们有新的快照可以下载了，我想下载最新的归档快照，这样是不是只会追加最近一段时间的区块数据；还是说，又是完整的下载一份快照数据？

又一份完整数据，我们暂时不提供增量数据下载支持

newbie-ksc commented 1 week ago

请教一个场景； 我下载了上一个版本的归档快照，然后开始同步；过了一段时间后，发现咱们有新的快照可以下载了，我想下载最新的归档快照，这样是不是只会追加最近一段时间的区块数据；还是说，又是完整的下载一份快照数据？

又一份完整数据，我们暂时不提供增量数据下载支持

好的，感谢解答