replicate / cog-llama-template

LLaMA Cog template
Apache License 2.0
307 stars 52 forks source link

use python downloader instead of pget #61

Closed technillogue closed 1 year ago

technillogue commented 1 year ago

this will reuse connections across downloaded files, which can't be done with current versions of pget. it uses a semaphore to limit the number of connections across all files without explicit batching or changing concurrency based on number or files

technillogue commented 1 year ago

Questions and some minor comments related to chunksize. None of these are blockers, but I want to shake out anything that could impact us trying to move away from pget.

unless we get loading from memory without going through disk, I'd go back to pget once we have multifile, this isn't long term (except for downloading loras)

technillogue commented 1 year ago

Is it worth making the downloader into a small Python package and putting it on PyPI?

it's already available as a separate package https://github.com/technillogue/pyget but I prefer keeping it in the same repo for speed of iteration