search

eth-sri / lmql

A language for constraint-guided and efficient LLM programming.
https://lmql.ai
Apache License 2.0
3.7k stars 200 forks source link

Support for Phi-2 #300

Closed mlinegar closed 8 months ago

mlinegar commented 11 months ago

Is it possible to use Microsoft's Phi models with LMQL, or are there any plans to make this possible? It seems to have been incorporated into llama.cpp (see here for example).

When I try to load the model, e.g. with:

lmql serve-model --loader auto-gptq --cuda --static /local/path/phi-2-GPTQ --disable_exllamav2 True --use_safetensors True --inject_fused_attention False --inject_fused_mlp False --trust_remote_code True

I get the error: phi-msft isn't supported yet.

Are there any plans to support this class of model?

lbeurerkellner commented 10 months ago

I have not tested phi-2 tokenizers with LMQL yet, but the error you seeing does not come from the LMQL runtime. It sounds more like the installed auto-gptq version does not yet support phi-2.