[bug] Unable to use Langchain + Pydantic

[yudhiesh]

Describe the bug I am referencing the documentation about the LangChain Integration which only showcases how it works using RAILSPEC and trying to make it work with Pydantic.

To Reproduce Steps to reproduce the behavior:

1. Setup Pydantic Model

   
   from pydantic import BaseModel, Field

class LLMResponse(BaseModel): generated_sql : str = Field(description="Generated SQL for the given natural language instruction.")

2. Create Prompt Template
```python 
import openai
from rich import print

from langchain.output_parsers import GuardrailsOutputParser

from langchain.prompts import PromptTemplate

ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE = """
    Generate PostgreSQL code based on the query. 
    Only generate PostgreSQL code and nothing else. 
    Explanations are not needed.
    Query: ${query}

    ${gr.complete_json_suffix_v2}
"""

prompt = PromptTemplate(
    template=ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE,
    input_variables=["query"],
)

3. Create Output Parser

   output_parser = GuardrailsOutputParser.from_pydantic(LLMResponse, api=openai.chat.completions.create, prompt=prompt)

4. Check Prompt is in Output Parser This is where the issue is happening, the following line prints nothing meaning that the prompt does not get injected into the output_parser when using Pydantic.

   print(output_parser.guard.prompt)

5. Perform Validation

   output = llm(prompt.format_prompt(query="Select the name of the employee who has the highest salary.").to_string())

   This throws the following error:

   
   ---------------------------------------------------------------------------

KeyError Traceback (most recent call last)

in <cell line: 1>() ----> 1 output = llm(prompt.format_prompt(query=SAMPLE_QUERY).to_string())

7 frames

/usr/lib/python3.10/string.py in get_value(self, key, args, kwargs) 225 return args[key] 226 else: --> 227 return kwargs[key] 228 229

KeyError: 'gr'

**Expected behavior**
The output from this:
```python 
print(output_parser.guard.prompt)

Should be:

PromptTemplate(
    input_variables=['gr.complete_json_suffix_v2', 'query'],
    template='\n    Generate PostgreSQL code based on the query. \n    Only generate PostgreSQL code and nothing 
else. \n    Explanations are not needed.\n    Query: ${query}\n    \n    ${gr.complete_json_suffix_v2}\n'
)

Library version:

Name: guardrails-ai
Version: 0.3.2
Summary: Adding guardrails to large language models.
Home-page: 
Author: Guardrails AI
Author-email: [contact@guardrailsai.com](mailto:contact@guardrailsai.com)
License: Apache-2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: eliot, eliot-tree, griffe, lxml, openai, pydantic, pydash, python-dateutil, regex, rich, rstr, tenacity, tiktoken, typer, typing-extensions

Additional context I have tried making some changes to the code as I noticed in GuardrailsOutputParser.from_pydantic() that the prompt is set by default to a "". I made a change to get it to reference the prompt as a kwarg

    @classmethod
    def from_pydantic(
        cls,
        output_class: Any,
        num_reasks: int = 1,
        api: Optional[Callable] = None,
        *args: Any,
        **kwargs: Any,
    ) -> GuardrailsOutputParser:
        try:
            from guardrails import Guard
        except ImportError:
            raise ImportError(
                "guardrails-ai package not installed. "
                "Install it by running `pip install guardrails-ai`."
            )
        return cls(
            guard=Guard.from_pydantic(output_class, kwargs.get("prompt"), num_reasks=num_reasks), # Added here
            api=api,
            args=args,
            kwargs=kwargs,
        )

This seems to fix it to a certain extent where now the output_parser

Prompt Template:

print(prompt)

PromptTemplate(
    input_variables=['gr.complete_json_suffix_v2', 'query'],
    template='\n    Generate PostgreSQL code based on the query. \n    Only generate PostgreSQL code and nothing 
else. \n    Explanations are not needed.\n    Query: ${query}\n\n    ${gr.complete_json_suffix_v2}\n'
)

Prompt from Output Parser:

output_parser = GuardrailsOutputParser.from_pydantic(LLMResponse, api=openai.chat.completions.create, prompt=prompt.template)
print(output_parser.guard.prompt)

Output from above:

Generate PostgreSQL code based on the query. 
    Only generate PostgreSQL code and nothing else. 
    Explanations are not needed.
    Query: ${query}

Given below is XML that describes the information to extract from this document and the tags to extract it into.

<output>
    <string name="generated_sql" description="Generated SQL for the given natural language instruction."/>
</output>

ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` 
attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON
MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and 
specific types. Be correct and concise.

Here are examples of simple (XML, JSON) pairs that show the expected behavior:
- `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}`
- `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}`
- `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" format="1-indexed" 
/></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}`

But when I try to validate using:

output = llm(prompt.format_prompt(query=SAMPLE_QUERY).to_string())

I get the following error:

---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

[<ipython-input-39-afcf7160ff57>](https://localhost:8080/#) in <cell line: 1>()
----> 1 output = llm(prompt.format_prompt(query=SAMPLE_QUERY).to_string())

7 frames

[/usr/lib/python3.10/string.py](https://localhost:8080/#) in get_value(self, key, args, kwargs)
    225             return args[key]
    226         else:
--> 227             return kwargs[key]
    228 
    229 

KeyError: 'gr'

Which seems to be caused by ${gr.complete_json_suffix_v2} not being picked up and updated with the JSON Schema prompt which is added via the RAILSpec.

[CalebCourier]

Hi @yudhiesh, I'm looking into this now and will get back to you once I've found something.

One thing to call out is that we still have a PR open with LangChain to update the GuardrailsOutputParser in order to support Guardrails >=0.3.x, so it may not work with the newer versions at the moment: https://github.com/langchain-ai/langchain/pull/14657

[CalebCourier]

@yudhiesh along with your updates to GuardrailsOutputParser.from_pydantic, have you tried specifying the prompt as we show in the integration docs?

This would look like:

ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE = """
    Generate PostgreSQL code based on the query. 
    Only generate PostgreSQL code and nothing else. 
    Explanations are not needed.
    Query: ${query}

    ${gr.complete_json_suffix_v2}
"""

output_parser = GuardrailsOutputParser.from_pydantic(LLMResponse, api=openai.chat.completions.create, prompt=ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE)

prompt = PromptTemplate(
    template=output_parser.guard.prompt.escape(),
    input_variables=output_parser.guard.prompt.variable_names,
)

output = llm(prompt.format_prompt(query="Select the name of the employee who has the highest salary.").to_string())

I believe you will still encounter the version mismatch issues address in the above referenced PR, but this should resolve the prompt formatting issue you're encountering.

[yudhiesh]

@yudhiesh along with your updates to GuardrailsOutputParser.from_pydantic, have you tried specifying the prompt as we show in the integration docs?

This would look like:

ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE = """

    Generate PostgreSQL code based on the query. 

    Only generate PostgreSQL code and nothing else. 

    Explanations are not needed.

    Query: ${query}

    ${gr.complete_json_suffix_v2}

"""

output_parser = GuardrailsOutputParser.from_pydantic(LLMResponse, api=openai.chat.completions.create, prompt=ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE)

prompt = PromptTemplate(

    template=output_parser.guard.prompt.escape(),

    input_variables=output_parser.guard.prompt.variable_names,

)

output = llm(prompt.format_prompt(query="Select the name of the employee who has the highest salary.").to_string())

I believe you will still encounter the version mismatch issues address in the above referenced PR, but this should resolve the prompt formatting issue you're encountering.

Lemme give it a try and get back to you.

[yudhiesh]

@yudhiesh along with your updates to GuardrailsOutputParser.from_pydantic, have you tried specifying the prompt as we show in the integration docs?

This would look like:

ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE = """
    Generate PostgreSQL code based on the query. 
    Only generate PostgreSQL code and nothing else. 
    Explanations are not needed.
    Query: ${query}

    ${gr.complete_json_suffix_v2}
"""

output_parser = GuardrailsOutputParser.from_pydantic(LLMResponse, api=openai.chat.completions.create, prompt=ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE)

prompt = PromptTemplate(
    template=output_parser.guard.prompt.escape(),
    input_variables=output_parser.guard.prompt.variable_names,
)

output = llm(prompt.format_prompt(query="Select the name of the employee who has the highest salary.").to_string())

I believe you will still encounter the version mismatch issues address in the above referenced PR, but this should resolve the prompt formatting issue you're encountering.

I downgraded to guardrails-ai==0.2.9 and tried the code but got the following error:

---------------------------------------------------------------------------

BadRequestError                           Traceback (most recent call last)

[<ipython-input-15-cbf49d77c939>](https://localhost:8080/#) in <cell line: 26>()
     24 )
     25 
---> 26 output = llm(prompt.format_prompt(query=SAMPLE_QUERY).to_string())

11 frames

[/usr/local/lib/python3.10/dist-packages/openai/_base_client.py](https://localhost:8080/#) in _request(self, cast_to, options, remaining_retries, stream, stream_cls)
    957 
    958             log.debug("Re-raising status error")
--> 959             raise self._make_status_error_from_response(err.response) from None
    960 
    961         return self._process_response(

BadRequestError: Error code: 400 - {'error': {'message': "[] is too short - 'messages'", 'type': 'invalid_request_error', 'param': None, 'code': None}}

[CalebCourier]

@yudhiesh my apologies, the example of constructing the prompt template I posted is for using the completions API. If you are using a chat model you will likely want to construct the prompt template as messages like this:

from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)
system_message_prompt = SystemMessagePromptTemplate.from_template(
    template=output_parser.guard.instructions.escape(),
    input_variables=output_parser.guard.instructions.variable_names,
)
human_message_prompt = HumanMessagePromptTemplate.from_template(
    template=output_parser.guard.prompt.escape(),
    input_variables=output_parser.guard.prompt.variable_names,
)

chat_prompt = ChatPromptTemplate.from_messages(
    [system_message_prompt, human_message_prompt]
)

Then when you call the llm, you'll likely want to use to_messages instead of to_string:

output = llm(chat_prompt.format_prompt(query="Select the name of the employee who has the highest salary.").to_messages())

[yudhiesh]

output = llm(chat_prompt.format_prompt(query="Select the name of the employee who has the highest salary.").to_messages())

Still throws an error, but this time due to the Guard object not having any instructions Guard(RAIL=Rail(input_schema=None, output_schema=JsonSchema(Object({'text': String({})})), instructions=None, prompt=Prompt(), version='0.1'))

system_message_prompt = SystemMessagePromptTemplate.from_template(
    template=output_parser.guard.instructions.escape(), # Throws an AttributeError
    input_variables=output_parser.guard.instructions.variable_names,
)

I guess for now I will not use Langchain with Guardrails until things are more stable and will fallback to the following code:

import openai
from rich import print
import guardrails as gd

ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE = """
    Generate a valid SQL query for the following natural language instruction:

    ${query}

    ${gr.complete_json_suffix_v2}
"""

guard = gd.Guard.from_pydantic(output_class=LLMResponse, prompt=ZERO_SHOT_PROMPT_GUARDRAILS_TEMPLATE)

raw_llm_output, validated_output, *rest = guard(
    llm_api=openai.chat.completions.create,
    model=MODEL_NAME,
    prompt_params={
        "query": SAMPLE_QUERY
    },
)

Just that I will need to handle retries to OpenAI myself or via another library for now as Langchain did that for me.

[yudhiesh]

@CalebCourier is this fixed in the following PR?

[CalebCourier]

@yudhiesh I would say that this issue is partially fixed by that PR which was released in guardrails 0.4.0. You should now be able to use any guard in a Langchain Expression Language chain (I'll include an example below). The caveat is that we do not yet support reasking with this pattern yet.

Example:

from rich import print
from langchain_core.output_parsers import JsonOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from guardrails import Guard
from guardrails.validators import ValidRange
from pydantic import BaseModel, Field

class Patient(BaseModel):
        gender: str = Field(description="Patient's gender")
        age: int = Field(validators=[ValidRange(min=0, max=100, on_fail="fix")])
        symptoms: str = Field(description="Symptoms that the patient is currently experiencing")

prompt = ChatPromptTemplate.from_template("""
You are a helpful assistant only capable of communicating with valid JSON, and no other text.

Given the following doctor's notes about a patient, please extract a dictionary that contains the patient's information.

{doctors_notes}

ONLY return a valid JSON object (no other text is necessary). Be correct and concise.

Here's an example:
```json
{{
    "gender": "Female",
    "age": 42,
    "symptoms": "hair loss, loss of appetite, sudden decrease in bodyweight"
}}

""")

doctors_notes = """ 49 y/o Male with chronic macular rash to face & hair, worse in beard, eyebrows & nares. Itchy, flaky, slightly scaly. Moderate response to OTC steroid cream """

model = ChatOpenAI(model="gpt-3.5-turbo")

guard = Guard.from_pydantic(output_class=Patient)

output_parser = JsonOutputParser()

chain = prompt | model | guard | output_parser

response = chain.invoke({ "doctors_notes": doctors_notes })

print("type(response): ", type(response)) print("response: ", response)

print(guard.history.last.tree)

[yudhiesh]

Thanks let me try that out! I would be more than happy to help out with the reasking logic but would need some assistance with understanding how it works.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity.