dvmazur / mixtral-offloading

Run Mixtral-8x7B models in Colab or consumer desktops
MIT License
2.29k stars 225 forks source link

FastAPI Integration and Performance Benchmarking #29

Open Jnmz opened 6 months ago

Jnmz commented 6 months ago

This PR addes a python script version (demo.py) of the original Jupyter notebook (notebooks/demo.ipynb). This transition facilitates a more streamlined deployment process, enabling the project's functionality to be accessed via a FastAPI interface. Additionally, the inclusion of a benchmarking script (benchmark.py) provides users with the tools to evaluate performance metrics effectively. These contributions collectively broaden the project's usability and provide a foundation for further development and integration into diverse environments.