search

eth-sri / lmql

A language for constraint-guided and efficient LLM programming.
https://lmql.ai
Apache License 2.0
3.48k stars 191 forks source link

Allow specifying model path for hf models #313

Closed Ranguna closed 4 months ago

Ranguna commented 5 months ago

The llama.cpp loader accepts a path for gguf models, but I was not able to find this option for models loaded with hf transfomers. I have quite a few models that I use with https://github.com/oobabooga/text-generation-webui/ and it's not ideal having to download them again just for lmql.

lbeurerkellner commented 5 months ago

You should be able to just specify the full (absolute) path as model identifier. Does that work for you?