search

dvmazur / mixtral-offloading

Run Mixtral-8x7B models in Colab or consumer desktops
MIT License
2.28k stars 223 forks source link

CLI interface added #12

Open NJannasch opened 6 months ago

NJannasch commented 6 months ago

This PR adds a small CLI interface to the repository which makes local usage easy.

dvmazur commented 6 months ago

Hey, @NJannasch! Thanks for the PR and sorry for the long reply. We would like to make sure the script works before merging it. Could you maybe make a Colab notebook that runs the script?

NJannasch commented 6 months ago

Hi @dvmazur, You can just checkout the branch, install dependencies and then call python3 cli.py as described in my README update.

An example Colab is available here (with some interestingly long responses lol): https://colab.research.google.com/drive/1Dkh0WNJAZD73Q4l4R0Y0T699mnAn6M5f?usp=sharing

Z-KN commented 3 months ago

I tried it out. This script works. Please merge if possible.