[BOUNTY - $300] Support Multi-GPU

exo-explore / exo

Run your own AI cluster at home with everyday devices 📱💻 🖥️⌚

GNU General Public License v3.0

10.29k stars 578 forks source link

[BOUNTY - $300] Support Multi-GPU #223

Open AlexCheema opened 3 weeks ago

AlexCheema commented 3 weeks ago

Currently you can only run one exo instance on each device.

There are some design decisions here:

Should we support running multiple exo instances on the same device, with one per GPU
Or should we support running one exo instance that uses multiple GPUs

Sean-fn commented 1 day ago

I agree with supporting one exo instance that uses multiple GPUs. This approach would allow us to shard more when only one model is inferencing. What do you think?