How to load in-process model from my local folder?

eth-sri / lmql

A language for constraint-guided and efficient LLM programming.

https://lmql.ai

Apache License 2.0

3.64k stars 197 forks source link

How to load in-process model from my local folder? #291

Closed SuperMasterBlasterLaser closed 9 months ago

SuperMasterBlasterLaser commented 9 months ago

Hello everyone.

I've downloaded one Transformers model into my local folder and trying to load it into lmql. This documentation page is very hard to understand, how I can provide path to my model folder. When I used guidance I was able to provide it via AutoTokenizer, AutoModelForCausalLM

How to load model from folder?

SuperMasterBlasterLaser commented 9 months ago

I think I figured out. I needed to launch it separately. I'm sorry.