I defined a JSON schema through Pydantic, along with an accompanying prompt, following the chat format for Phi-3 mini.
I then generate an output from the model, but it comes up empty, after a considerable time (over 1 minute). I would like to see what the library is doing during this time, in order to debug my program. I assume that it attempts
to generate valid JSON, fails, and falls back to returning an empty object. Would that be a fair assessment?
Steps/code to reproduce the bug:
import outlines
from pydantic import BaseModel
CONTEXT_SIZE = 4096
hf_models_path = Path().home() / "hf-models"
model_path = (
hf_models_path / "Phi-3-mini-4k-instruct-gguf" / "Phi-3-mini-4k-instruct-q4.gguf"
)
llm = Llama(str(model_path), n_ctx=CONTEXT_SIZE, verbose=False)
model = models.LlamaCpp(llm)
class Item(BaseModel):
item: str
description: str
class Step(BaseModel):
step: str
description: str
items: list[Item]
class Recipe(BaseModel):
steps: list[Step]
# use phi-3 chat template
@outlines.prompt
def convert_recipe(recipe):
"""<|user|>
Convert the following recipe into a structured JSON object.
{{ recipe }}<|end|>
<|assistant|>"""
generator = outlines.generate.json(model, Recipe)
with open("recipe.txt") as f:
recipe = f.read()
prompt = convert_recipe(recipe)
result = generator(convert_recipe(recipe), max_tokens=4096)
Expected result:
`result` to contain a full object following the schema.
Describe the issue as clearly as possible:
I defined a JSON schema through Pydantic, along with an accompanying prompt, following the chat format for Phi-3 mini. I then generate an output from the model, but it comes up empty, after a considerable time (over 1 minute). I would like to see what the library is doing during this time, in order to debug my program. I assume that it attempts to generate valid JSON, fails, and falls back to returning an empty object. Would that be a fair assessment?
Steps/code to reproduce the bug:
Expected result:
Error message:
Outlines/Python version information:
Version information
Context for the issue:
No response