Evaluate TheBloke/Phind-CodeLlama-34B-v2-AWQ

[the-crypt-keeper]

Latest vLLM adds AWQ support, it would be interesting to compare performance vs the native awq executor we already support:

• pull up AWQ to latest
• pull up vLLM to latest
• try this model with both standalone and vLLM

[the-crypt-keeper]

its working through vLLM but it turns out this is a new kind of AWQ:

https://github.com/casper-hansen/AutoAWQ

these models are not working with even the latest version of the old awq repo:

RuntimeError: shape '[1, 29, 1024]' is invalid for input of size 237568

So looks like will have to implement autoawq to replace the legacy awq.

[the-crypt-keeper]

Completed via vllm-awq.