Support for Quantized MIXTRAL 8x7B models in sparse / base mode

Alpha-VLLM / LLaMA2-Accessory

An Open-source Toolkit for LLM Development

https://llama2-accessory.readthedocs.io/

Other

2.73k stars 176 forks source link

Support for Quantized MIXTRAL 8x7B models in sparse / base mode #142

Open PlanetMacro opened 10 months ago

PlanetMacro commented 10 months ago

Would it be possible to reformat the Quantized models of MIXTRAL 8x7B to run in sparse /base mode in LLaMA2-Accessory?

ChrisLiu6 commented 10 months ago

Theoretically all mixtral implementations are interchangeable. However, since I don't know what your Quantized model looks like, I am not able to suggest the specific pipeline

PlanetMacro commented 10 months ago

https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF