[BOUNTY - $100] Parallelise Model Loading

exo-explore / exo

Run your own AI cluster at home with everyday devices 📱💻 🖥️⌚

GNU General Public License v3.0

10.97k stars 639 forks source link

Open AlexCheema opened 2 months ago

AlexCheema commented 2 months ago

Right now we already parallelise model downloads which works great and speeds things up a lot
However, loading the model into memory can also be slow for large models, and right now exo will do this sequentially one node at a time
In a similar way to model downloads, we should parallelise model loading into memory
This should probably expose some functionality of ensure_shard in the abstract InferenceEngine, maybe something like preload_model

aybanda commented 1 month ago

@AlexCheema submitted PR https://github.com/exo-explore/exo/pull/211 Hope it resolves your issue

If you feel like supporting me:

vovw commented 2 weeks ago

tried to take a turn at this #360 lmk if there are any req changes