caleb-kaiser
commented
6 days ago @syuoni I've run into similar problems when using smaller models with the JSON grammar + greedy sampling. Using the JSON grammar defined in https://github.com/dottxt-ai/outlines/blob/main/outlines/grammars/json.lark , we can see that your output string is technically valid:
from lark import Lark
grammar = r"""
?start: value
?value: object
| array
| ESCAPED_STRING
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
array : "[" [value ("," value)*] ["]"]
object : "{" [pair ("," pair)*] ["}"]
pair : ESCAPED_STRING ":" value
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""
parser = Lark(grammar, start="start", debug=True)
parser.parse("000000000000000000000000000000")
# Tree('number', [Token('SIGNED_NUMBER', '0000000000000000000000000000')])Smaller models, in my experience, are more prone to this kind of output, particularly when we only sample the most likely token.
Have you tried the generate.json() method instead? I just tested the following (rolling back to outlines 0.1.3 due to a bug in 0.1.4) with your prompt and it seems to work:
from pydantic import BaseModel
class Answer(BaseModel):
answer: int
generator = outlines.generate.json(model, Answer)
generator(prompt)
# Answer(answer=1)
Describe the issue as clearly as possible:
Using
outlines.grammars.jsondoes not make the outputs amenable to json format.Steps/code to reproduce the bug:
Expected result:
Error message:
No response
Outlines/Python version information:
Version information
Context for the issue:
No response