[Bug]: LLMTextCompletionProgram outputs correct answer, but causes ValueError

[jakubstrawa1]

Bug Description

When using LLMTextCompletionProgram to incorporate the pydantic modelling for my output i get a very strange result. The error indicates that the output is proper (134436839ABCDFG), but it still throws an error that it cannot extract json string from output. There's no way to get the values, because i have to call the llm_completion() method and it throws an error.

LLM im using is gpt 3.5 turbo

prompt_template =  """You are an AI expert in extracting data from documents. Below is a document extract from a Know Your Customer document. Consider this extract as context to the prompt and answer the below question. Question: {question} Context: {context}."""

question= "Extract the 'ABC' for the company from the text. The 'ABC' should be in the format XXXXXXXXXXXXXXXXXXXX, where X can be either a number or an uppercase letter (e.g., 529900QF6F5IB9WQCL03). Please respond with only the LEI and no additional information. Return 'No info' if no ABC is found."

prompt= json_val['prompts']['prompt_template'].format(question=question, context=processed_document)
llm_completion = LLMTextCompletionProgram.from_defaults(
    llm=llm,
    output_cls=ABC,
    prompt_template_str=prompt,
    verbose=True,
)

x = llm_completion()

class ABC(BaseModel):

    abc: str = Field(..., description="The 'ABC' or 'No info' if it doesn't exist.")

    @validator("abc")
    def validate_abc_code(cls, v):
        if v.lower() != "no info" and not re.match(r"^[0-9]{4}[0]{2}[A-Z0-9]{12}[0-9]{2}$", v):
            raise ValueError("Invalid ABC code format")
        return v

Version

0.10.54

Steps to Reproduce

I tested it on gpt 3.5 llm. The processed file is a string of a random document that im trying to extract information from.

Relevant Logs/Tracbacks

ValueError: Could not extract json string from output: "abc": "134436839ABCDFG"
File <command-1015110231477814>, line 1
----> 1 llm_completion()
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/llama_index/core/output_parsers/utils.py:112, in extract_json_str(text)
    110 match = re.search(r"\{.*\}", text.strip(), re.MULTILINE | re.IGNORECASE | re.DOTALL)
    111 if not match:
--> 112     raise ValueError(f"Could not extract json string from output: {text}")
    114 return match.group()

[dosubot[bot]]

Hey @jakubstrawa1! 👋 I'm here to help you with any bugs, questions, or contributions you have. Let's squash those bugs together!

To resolve the error indicating that a JSON string cannot be extracted from the output when using LLMTextCompletionProgram with Pydantic modeling, ensure that the output from the guidance program is correctly formatted as JSON and wrapped in a markdown block. Here is a detailed approach to address this issue:

1. Ensure JSON Block Formatting: Make sure the response from the LLM is a JSON block wrapped in markdown formatting (i.e., enclosed within json and).

2. Parsing Function: Use the parse_pydantic_from_guidance_program function to parse the output. This function attempts to parse the last markdown formatted JSON block from the response.

Here is the relevant part of the code that handles this:

from typing import Type, TypeVar
from llama_index.core.bridge.pydantic import BaseModel
from llama_index.core.output_parsers.base import OutputParserException
from llama_index.core.output_parsers.utils import parse_json_markdown

Model = TypeVar("Model", bound=BaseModel)

def parse_pydantic_from_guidance_program(
    response: str, cls: Type[Model], verbose: bool = False
) -> Model:
    try:
        output = response.split("```json")[-1]
        output = "```json" + output
        if verbose:
            print("Raw output:")
            print(output)
        json_dict = parse_json_markdown(output)
        sub_questions = cls.parse_obj(json_dict)
    except Exception as e:
        raise OutputParserException(
            "Failed to parse pydantic object from guidance program"
            ". Probably the LLM failed to produce data with right json schema"
        ) from e
    return sub_questions

3. Handling JSON Formatting Issues: The parse_json_markdown function includes additional logic to handle potential issues with JSON formatting, such as trailing commas.

Here is the relevant part of the parse_json_markdown function:

import contextlib
import json
import re
from typing import Any, List

with contextlib.suppress(ImportError):
    import yaml

from llama_index.core.output_parsers.base import OutputParserException

def _marshal_llm_to_json(output: str) -> str:
    output = output.strip().replace("{{", "{").replace("}}", "}")

    left_square = output.find("[")
    left_brace = output.find("{")

    if left_square < left_brace and left_square != -1:
        left = left_square
        right = output.rfind("]")
    else:
        left = left_brace
        right = output.rfind("}")

    return output[left : right + 1]

def parse_json_markdown(text: str) -> Any:
    if "```json" in text:
        text = text.split("```json")[1].strip().strip("```").strip()

    json_string = _marshal_llm_to_json(text)

    try:
        json_obj = json.loads(json_string)
    except json.JSONDecodeError as e_json:
        try:
            json_obj = yaml.safe_load(json_string)
        except yaml.YAMLError as e_yaml:
            raise OutputParserException(
                f"Got invalid JSON object. Error: {e_json} {e_yaml}. "
                f"Got JSON string: {json_string}"
            )
        except NameError as exc:
            raise ImportError("Please pip install PyYAML.") from exc

    return json_obj

Ensure that your LLM's output is correctly formatted as a JSON block within markdown, and the parse_pydantic_from_guidance_program function will handle the extraction and parsing of the JSON string into a Pydantic model ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[jakubstrawa1]

@dosu This does not work. I dont see how i can get the response that is required for parse_pydantic_from_guidance_program from the LLMTextCompletionProgram.

[logan-markewich]

@jakubstrawa1 the result has no brackets, so it cannot be parsed into JSON

[jakubstrawa1]

@logan-markewich That's really interesting, because i have 10 pydantic classes, and with llm.complete (without using pydantic), only the prompt and llm i get proper outputs. But then when i use LLMTextCompletionProgram i get errors that they cannot be parsed. All the inputs are the same and in the same format (str)

[jakubstrawa1]

And also there isn't any possiblity to see what response we got back from llm using LLMTextCompletionProgram.from_defaults() i can only presume that llm.complete(prompt) will give me approximately the same answer. So how can i know if its sent with or without brackets.