search

bigscience-workshop / petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
https://petals.dev
MIT License
8.89k stars 490 forks source link

Is there a way to shard a model without downloading it first? #568

Closed nrs-status closed 3 months ago

nrs-status commented 3 months ago

I'd like to run a private swarm on VMs will low disk storage, is there any way I could split a model and manually select which shard goes into which VM?

tonywang16 commented 3 months ago

You could download the whole model on one VM then use NFS share the model folder to other VMs.

nrs-status commented 3 months ago

@tonywang16 Thanks, this definitely works!