search

exo-explore / exo

Run your own AI cluster at home with everyday devices 📱💻 🖥️⌚
GNU General Public License v3.0
6.29k stars 322 forks source link

[BOUNTY - $200] Batched Requests #1

Open AlexCheema opened 1 month ago

AlexCheema commented 1 month ago

Motivation: Batching multiple inference requests together can speed up inference. Batching can even be leveraged with single-input settings for speedups with e.g. staged speculative decoding.

What: Currently, exo handles inference requests separately. This bounty is for batching inferences together, so that multiple inputs can be passed through model shards together in a single pass.

Reward: $200 Bounty paid out with USDC on Ethereum, email alex@exolabs.net

abdussamettrkr commented 1 month ago

Working on this

AlexCheema commented 1 month ago

@abdussamettrkr drop a comment here if you need any help / run into any issues. here to help

githubbin765 commented 4 days ago

Is this still available? Are there also other bounties? Are they open to anyone?