Closed technillogue closed 1 year ago
Questions and some minor comments related to chunksize. None of these are blockers, but I want to shake out anything that could impact us trying to move away from pget.
unless we get loading from memory without going through disk, I'd go back to pget once we have multifile, this isn't long term (except for downloading loras)
Is it worth making the downloader into a small Python package and putting it on PyPI?
it's already available as a separate package https://github.com/technillogue/pyget but I prefer keeping it in the same repo for speed of iteration
this will reuse connections across downloaded files, which can't be done with current versions of pget. it uses a semaphore to limit the number of connections across all files without explicit batching or changing concurrency based on number or files