Will this work with quantized versions of llama models?

[thistleknot]

What about alpaca?

For example. I have a smaller version of llama that runs locally

python server.py --model ggml-alpaca-7b-q4 --listen

I'm going to give it a shot/go with this reduced model and get back to you over the weekend