pg9182
commented
2 years ago The things I considered while writing the new script:
- Compatibility with other container registries.
- Cleanup on failure, resuming downloads, checksum verification without reading stuff twice.
- Proper file timestamps.
- Fewer dependencies (especially since parallel pulls in a bunch of junk on most distros).
- POSIX sh and busybox compatibility.
Current script by pg: https://discord.com/channels/920776187884732556/930743666761429042/942501682347122758
Currently implemented script: https://github.com/l1ghthouse/northstar-bot/blob/78495818bc9654a50d6b5af7523eef7ab2971b79/src/providers/util/util.go#L104
At the moment, it seems like the script that runs in parallel works a bit faster, but could result in timeouts, and rate limits (though, I haven't run into them, since IPs are randomized)
Look into getting a faster download, with backoffs to handle timeouts.