search

ChainSafe / lodestar

🌟 TypeScript Implementation of Ethereum Consensus
https://lodestar.chainsafe.io
Apache License 2.0
1.17k stars 289 forks source link

Improve experience of `download-spec-tests` #6991

Open ensi321 opened 2 months ago

ensi321 commented 2 months ago

Problem description

As the size of spec test files increases in every release (from ~670MB for v1.4.0 to ~780MB for v1.5.0-alpha.3), it takes 20+ minutes for me to download all files.

Currently downloading spec test provides minimal log output on the download progress. Only at the beginning of download, and when the download is finished, which could be 10-20 minutes apart. It would be great to provide more info (eg. a progress bar, download time remaining) during download such that users/devs know what to expect. Here is a sample log of the current behaviour:

NC@Chainsafe beacon-node % yarn download-spec-tests
yarn run v1.22.22
$ node --loader=ts-node/esm test/spec/downloadTests.ts
outputDir = /Users/NC/Documents/lodestar/packages/beacon-node/spec-tests
Downloading new version v1.4.0-beta.6
Cleaning existing version v1.5.0-alpha.3 at /Users/NC/Documents/lodestar/packages/beacon-node/spec-tests
outputDir = /Users/NC/Documents/lodestar/packages/beacon-node/spec-tests-bls
version v0.1.1 already downloaded
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/general.tar.gz - 15889101 bytes
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz - 421709376 bytes
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes
Downloaded  https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/general.tar.gz
Download attempt 2 for https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz failed: aborted
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz - 421709376 bytes
Downloaded  https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz
Download attempt 2 for https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz failed: aborted
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes
Download attempt 3 for https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz failed: aborted
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes
Downloaded https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes

Moreover, partially downloaded files are deleted when reattempting to download spec test. This is an issue when having slow and unstable internet, a user may already have downloaded general and minimal spec tests, but internet drops when downloading mainnet. When he runs yarn download-spec-tests again, general and minimal are deleted and everything is being re-downloaded.

Solution description

Additional context

No response

wemeetagain commented 2 months ago

definitely agree, I've run into the same problem