OutputFixingParser not working

zhuohanl commented 3 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from dotenv import load_dotenv, find_dotenv

from langchain_openai import ChatOpenAI
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.output_parsers import PydanticOutputParser
from langchain.output_parsers import OutputFixingParser
from langchain.prompts import PromptTemplate

_ = load_dotenv(find_dotenv())

llm = ChatOpenAI(model="gpt-4o")

##############################
### Auto-Fixing Parser
##############################

class Date(BaseModel):
    year: int = Field(description="Year")
    month: int = Field(description="Month")
    day: int = Field(description="Day")
    era: str = Field(description="BC or AD")

prompt_template = """
Extact the date within user input.
{format_instructions}
User Input:
{query}
"""

parser = PydanticOutputParser(pydantic_object=Date)
new_parser = OutputFixingParser.from_llm(
    parser=parser,
    llm=llm
)

template = PromptTemplate(
    template=prompt_template,
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)

query = "Sunny weather on April 6 2023"
prompt = template.format_prompt(query=query)

response = llm.invoke(prompt.to_messages())

incorrect_output = response.content.replace("4", "April")

print("====Incorrect output=====")
print(incorrect_output) 

try:
    response = parser.parse(incorrect_output)
except Exception as e:
    print("===Exception===")
    print(e)

    print("===Parsing using outputfixingparser===")
    date = new_parser.parse(incorrect_output)
    print(date.json())

Error Message and Stack Trace (if applicable)

Traceback (most recent call last):
  File "/Users/selinali/Documents/dev/zhihu-ai-engineer-lecture-notes/practise/08-langchain/run2.py", line 60, in <module>
    date = new_parser.parse(incorrect_output)
  File "/Users/selinali/Documents/dev/zhihu-ai-engineer-lecture-notes/.venv/lib/python3.10/site-packages/langchain/output_parsers/fix.py", line 69, in parse
    return self.parser.parse(completion)
  File "/Users/selinali/Documents/dev/zhihu-ai-engineer-lecture-notes/.venv/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py", line 77, in parse
    return super().parse(text)
  File "/Users/selinali/Documents/dev/zhihu-ai-engineer-lecture-notes/.venv/lib/python3.10/site-packages/langchain_core/output_parsers/json.py", line 98, in parse
    return self.parse_result([Generation(text=text)])
  File "/Users/selinali/Documents/dev/zhihu-ai-engineer-lecture-notes/.venv/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Generation
text
  str type expected (type=type_error.str)

Description

I am trying to test using OutputFixingParser by following a tutorial. But it gave me exception as shown in the Stack Trace.

System Info

python = "^3.10" langchain = "^0.2.11" langchain-openai = "^0.1.19"

eyurtsev commented 3 months ago

What's the output of python -m langchain_core.sys_info

isahers1 commented 3 months ago

Should be solved by: https://github.com/langchain-ai/langchain/pull/24687

Once the next langchain package version is released, this fix should be applied. In order to download the latest release, just run pip install -U langchain.

langchain-ai / langchain