Add Mixtral models - Githubissues

bigscience-workshop / petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

https://petals.dev

MIT License

8.89k stars 490 forks source link

Closed artek0chumak closed 3 months ago

artek0chumak commented 5 months ago

Add Mixtral models as runnable models.

TODO

[x] Wait for support of the new version of transformers (>4.35)
- [x] Add normal support for Cache
[x] Check compatibility of larger models (TinyMixtral, Mixtral-8x7B) on GPU
[x] Refactor some code(see # TODO)
[ ] Optimize code for improved speed (mb next PR?)