Open MiyazonoKaori opened 7 months ago
Yes! This is our top priorities.
Is there currently any work happening here? I could look into it if not.
context-free grammar
it would be cool if the llama cpp GBNF grammars could be standardize so we can do stuff like the image , between different LLM frameworks.
In our roadmap, anything https://github.com/outlines-dev/outlines supports can be part of vLLM, as long as any other frameworks that support LogitsProcessors API.
This looks good, just read through outlines - seems super useful.
When do we get it :D
Have someone done benchmark on latency introduced by outlines?
Is this already supported as described here https://github.com/vllm-project/vllm/issues/1191?
I would also love to see this feature make it into vllm!
Is this already supported as described here #1191?
I suspect the same.
Specifically https://github.com/vllm-project/vllm/pull/3211 implements handling for "request_format": { "type": "json_object" }
in the completion request body, which is what I'm familiar with when we say "JSON mode".
"request_format": { "type": "json_object" }
seems to work for me. Also requires a prompt specifically asking for json response.
Any plan to integrate modules such as lm-format-enforcer. Support JSON mode.