search

noamgat / lm-format-enforcer

Enforce the output format (JSON Schema, Regex etc) of a language model
MIT License
1.01k stars 46 forks source link

Oobabooga Extension for lm-format-enforcer #51

Open ResearchTLDR opened 6 months ago

ResearchTLDR commented 6 months ago

I used lm-format-enforcer with with llama-cpp-python example and adapted it a bit to get it to work, but I was limited to just what llama-cpp-python can handle. I have switched to using the Oobabooga text-generation-webui and its API. I cannot figure out how to use lm-format-enforcer with Oobabooga, though.

Is there some way to make lm-format-enforcer into an Extension for Oobabooga? That way it could be used both within the WebUI and also from the API. There is a JSON format extension here on Github, but it seems to have been abandoned months ago and when I just tried it, it did not limit output format to JSON at all: oobabooga-jsonformer-plugin

noamgat commented 6 months ago

This looks possible. Their example extension applies a logits processor, which is what LMFE is. I do not have the time to implement it right now. If you want to give it a shot I'll be happy to help. In the meantime, I'll leave this issue open, flag it as an enhancement, and use it for voting to see how popular this request is. Feel free to spread it around to get people to vote for it :)

unoriginalscreenname commented 6 months ago

@ResearchTLDR same! The jsonformer plugin seems broken. I'll visit the ooba example extension, but can't make any promises I'll figure that out.