smartcontracts / simple-optimism-node

The easiest way to run an Optimism node
MIT License
326 stars 112 forks source link

perf(utils): utilize axel for faster, multi-threaded downloads (#92) #93

Closed imtipi closed 4 months ago

imtipi commented 1 year ago

currently, when I download snapshot from Cloudflare, I have an unstable connection and can't download it at full speed, and I use axel can mitigate the issue. download progress command changed to du -b because axel will generate a .st file for continue download the file

mkostrikin commented 1 year ago

axel looks good, but very verbosibe.

[ 24%]  .......... .......... .......... .......... ..........  [48311.5KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48312.6KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48313.8KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48313.2KB/s]

While only this should report about progress.

while true ; do sleep 60; echo...
imtipi commented 1 year ago

axel looks good, but very verbosibe.

sometimes I found the downloading file has incorrect file size, like download 10secs but the file size has already been 200G in the disk, I can't add -q to disable axel status output until I can figure out this issue

smartcontracts commented 11 months ago

Merged another fix that uses aria2c. Can you see if that works well? If it's good enough then I'll use that, otherwise will explore using a multi-threaded downloader.

imtipi commented 4 months ago

close due to already switch to snap sync and arid2c