Open jakubstrawa1 opened 1 month ago
Hey @jakubstrawa1! 👋 I'm here to help you with any bugs, questions, or contributions you have. Let's squash those bugs together!
To resolve the error indicating that a JSON string cannot be extracted from the output when using LLMTextCompletionProgram
with Pydantic modeling, ensure that the output from the guidance program is correctly formatted as JSON and wrapped in a markdown block. Here is a detailed approach to address this issue:
Ensure JSON Block Formatting: Make sure the response from the LLM is a JSON block wrapped in markdown formatting (i.e., enclosed within json and
).
Parsing Function: Use the parse_pydantic_from_guidance_program
function to parse the output. This function attempts to parse the last markdown formatted JSON block from the response.
Here is the relevant part of the code that handles this:
from typing import Type, TypeVar
from llama_index.core.bridge.pydantic import BaseModel
from llama_index.core.output_parsers.base import OutputParserException
from llama_index.core.output_parsers.utils import parse_json_markdown
Model = TypeVar("Model", bound=BaseModel)
def parse_pydantic_from_guidance_program(
response: str, cls: Type[Model], verbose: bool = False
) -> Model:
try:
output = response.split("```json")[-1]
output = "```json" + output
if verbose:
print("Raw output:")
print(output)
json_dict = parse_json_markdown(output)
sub_questions = cls.parse_obj(json_dict)
except Exception as e:
raise OutputParserException(
"Failed to parse pydantic object from guidance program"
". Probably the LLM failed to produce data with right json schema"
) from e
return sub_questions
parse_json_markdown
function includes additional logic to handle potential issues with JSON formatting, such as trailing commas.Here is the relevant part of the parse_json_markdown
function:
import contextlib
import json
import re
from typing import Any, List
with contextlib.suppress(ImportError):
import yaml
from llama_index.core.output_parsers.base import OutputParserException
def _marshal_llm_to_json(output: str) -> str:
output = output.strip().replace("{{", "{").replace("}}", "}")
left_square = output.find("[")
left_brace = output.find("{")
if left_square < left_brace and left_square != -1:
left = left_square
right = output.rfind("]")
else:
left = left_brace
right = output.rfind("}")
return output[left : right + 1]
def parse_json_markdown(text: str) -> Any:
if "```json" in text:
text = text.split("```json")[1].strip().strip("```").strip()
json_string = _marshal_llm_to_json(text)
try:
json_obj = json.loads(json_string)
except json.JSONDecodeError as e_json:
try:
json_obj = yaml.safe_load(json_string)
except yaml.YAMLError as e_yaml:
raise OutputParserException(
f"Got invalid JSON object. Error: {e_json} {e_yaml}. "
f"Got JSON string: {json_string}"
)
except NameError as exc:
raise ImportError("Please pip install PyYAML.") from exc
return json_obj
Ensure that your LLM's output is correctly formatted as a JSON block within markdown, and the parse_pydantic_from_guidance_program
function will handle the extraction and parsing of the JSON string into a Pydantic model [1][2][3][4][5].
To continue talking to Dosu, mention @dosu.
@dosu This does not work. I dont see how i can get the response that is required for parse_pydantic_from_guidance_program from the LLMTextCompletionProgram.
@jakubstrawa1 the result has no brackets, so it cannot be parsed into JSON
@logan-markewich That's really interesting, because i have 10 pydantic classes, and with llm.complete (without using pydantic), only the prompt and llm i get proper outputs. But then when i use LLMTextCompletionProgram i get errors that they cannot be parsed. All the inputs are the same and in the same format (str)
And also there isn't any possiblity to see what response we got back from llm using LLMTextCompletionProgram.from_defaults() i can only presume that llm.complete(prompt) will give me approximately the same answer. So how can i know if its sent with or without brackets.
Bug Description
When using LLMTextCompletionProgram to incorporate the pydantic modelling for my output i get a very strange result. The error indicates that the output is proper (134436839ABCDFG), but it still throws an error that it cannot extract json string from output. There's no way to get the values, because i have to call the llm_completion() method and it throws an error.
LLM im using is gpt 3.5 turbo
Version
0.10.54
Steps to Reproduce
I tested it on gpt 3.5 llm. The processed file is a string of a random document that im trying to extract information from.
Relevant Logs/Tracbacks