search

the-crypt-keeper / can-ai-code

Self-evaluating interview for AI coders
https://huggingface.co/spaces/mike-ravkine/can-ai-code-results
MIT License
525 stars 30 forks source link

2bit Mixtral-8x7B with HQQ #126

Closed the-crypt-keeper closed 9 months ago

the-crypt-keeper commented 10 months ago

mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-attn-4bit-moe-2bit-HQQ

https://github.com/mobiusml/hqq

No transformers, custom runtime.

the-crypt-keeper commented 9 months ago

Completed, decent results but degradation is noticable. I managed only to get the non-compile pytorch backend to worked on an A100, inference was slow. Did not try the custom C++ backend.