search

noamgat / lm-format-enforcer

Enforce the output format (JSON Schema, Regex etc) of a language model
MIT License
1.53k stars 69 forks source link

Oobabooga Extension for lm-format-enforcer #51

Open ResearchTLDR opened 10 months ago

ResearchTLDR commented 10 months ago

I used lm-format-enforcer with with llama-cpp-python example and adapted it a bit to get it to work, but I was limited to just what llama-cpp-python can handle. I have switched to using the Oobabooga text-generation-webui and its API. I cannot figure out how to use lm-format-enforcer with Oobabooga, though.

Is there some way to make lm-format-enforcer into an Extension for Oobabooga? That way it could be used both within the WebUI and also from the API. There is a JSON format extension here on Github, but it seems to have been abandoned months ago and when I just tried it, it did not limit output format to JSON at all: oobabooga-jsonformer-plugin

noamgat commented 10 months ago

This looks possible. Their example extension applies a logits processor, which is what LMFE is. I do not have the time to implement it right now. If you want to give it a shot I'll be happy to help. In the meantime, I'll leave this issue open, flag it as an enhancement, and use it for voting to see how popular this request is. Feel free to spread it around to get people to vote for it :)

unoriginalscreenname commented 10 months ago

@ResearchTLDR same! The jsonformer plugin seems broken. I'll visit the ooba example extension, but can't make any promises I'll figure that out.