search

the-crypt-keeper / can-ai-code

Self-evaluating interview for AI coders
https://huggingface.co/spaces/mike-ravkine/can-ai-code-results
MIT License
525 stars 30 forks source link

Support AQLM #161

Closed the-crypt-keeper closed 2 months ago

the-crypt-keeper commented 8 months ago

https://github.com/Vahe1994/AQLM

the-crypt-keeper commented 8 months ago

Fairly easy to get going and Llama-7b works, but there's no Llama-Chat-* quants so I can't eval. Wasn't able to get the Mixtral-Instruct quant working. Revisit this in a few weeks..

the-crypt-keeper commented 2 months ago

AQLM is working with latest vLLM but performance is hillariously bad, on A100-40G I see 7 tok/sec on ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16