outlines-dev / outlines

Structured Text Generation
https://outlines-dev.github.io/outlines/
Apache License 2.0
8.14k stars 411 forks source link

JSON schema yields empty results using Phi-3-mini #1014

Open mwesthelle opened 2 months ago

mwesthelle commented 2 months ago

Describe the issue as clearly as possible:

I defined a JSON schema through Pydantic, along with an accompanying prompt, following the chat format for Phi-3 mini. I then generate an output from the model, but it comes up empty, after a considerable time (over 1 minute). I would like to see what the library is doing during this time, in order to debug my program. I assume that it attempts to generate valid JSON, fails, and falls back to returning an empty object. Would that be a fair assessment?

Steps/code to reproduce the bug:

import outlines
from pydantic import BaseModel

CONTEXT_SIZE = 4096

hf_models_path = Path().home() / "hf-models"
model_path = (
    hf_models_path / "Phi-3-mini-4k-instruct-gguf" / "Phi-3-mini-4k-instruct-q4.gguf"
)
llm = Llama(str(model_path), n_ctx=CONTEXT_SIZE, verbose=False)
model = models.LlamaCpp(llm)

class Item(BaseModel):
    item: str
    description: str

class Step(BaseModel):
    step: str
    description: str
    items: list[Item]

class Recipe(BaseModel):
    steps: list[Step]

# use phi-3 chat template
@outlines.prompt
def convert_recipe(recipe):
    """<|user|>
    Convert the following recipe into a structured JSON object.

    {{ recipe }}<|end|> 
    <|assistant|>"""

generator = outlines.generate.json(model, Recipe)

with open("recipe.txt") as f:
    recipe = f.read()

prompt = convert_recipe(recipe)
result = generator(convert_recipe(recipe), max_tokens=4096)

Expected result:

`result` to contain a full object following the schema.

Error message:

`result` yields `steps=[]`.

Outlines/Python version information:

Version information

``` 0.0.46 ```

Context for the issue:

No response

atapin commented 1 month ago

@mwesthelle could you add the recipe.txt file as well?