smartcontracts / simple-optimism-node

The easiest way to run an Optimism node
MIT License
336 stars 115 forks source link

perf(utils): utilize axel for faster, multi-threaded downloads (#92) #93

Closed imtipi closed 6 months ago

imtipi commented 1 year ago

currently, when I download snapshot from Cloudflare, I have an unstable connection and can't download it at full speed, and I use axel can mitigate the issue. download progress command changed to du -b because axel will generate a .st file for continue download the file

mkostrikin commented 1 year ago

axel looks good, but very verbosibe.

[ 24%]  .......... .......... .......... .......... ..........  [48311.5KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48312.6KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48313.8KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48313.2KB/s]

While only this should report about progress.

while true ; do sleep 60; echo...
imtipi commented 1 year ago

axel looks good, but very verbosibe.

sometimes I found the downloading file has incorrect file size, like download 10secs but the file size has already been 200G in the disk, I can't add -q to disable axel status output until I can figure out this issue

smartcontracts commented 1 year ago

Merged another fix that uses aria2c. Can you see if that works well? If it's good enough then I'll use that, otherwise will explore using a multi-threaded downloader.

imtipi commented 6 months ago

close due to already switch to snap sync and arid2c