Open servient-ashwin opened 1 month ago
The problem is with guided_json
and the way that is handled it seems, because without applying guided_json
the output is just fine.
Can confirm,
Changing the guided_decoding_backend
doesn't work either.
The model works well without the guided
guidance.
Your current environment
🐛 Describe the bug
I am using
vLLM
andNemo Instruct
and playing around with the new model (fp8) and even with vLLMs'guided_json
and including the schema in prompt, nemo almost never returns a output if JSON is involved however it does finish generating a output that is usually a letter or a symbol at the most.Does Mistral Nemo not support JSON output with vLLM? This works as expected with instruct-v0.3
This is what I get using the inference API on the hugging face model card
write a sample JSON response for a user info query,