.with_structured_output fails with `Key 'allOf' is not supported in schema, ignoring`

[antti-ngp]

There's something wrong with the json schema parsing when trying to extract structured output with ChatVertexAI.

Steps to reproduce:

from langchain_google_vertexai import ChatVertexAI
from langchain_core.pydantic_v1 import BaseModel, Field
from enum import Enum

class MyEnum(str,Enum):
    foo = 'Foo'
    bar = 'Bar'

class MyOutput(BaseModel):
    my_field: MyEnum = Field(...,description="This is my field")

MyOutput.schema_json()
# {
#     "title": "MyOutput",
#     "type": "object",
#     "properties": {
#         "my_field": {
#             "description": "This is my field",
#             "allOf": [
#                 {
#                     "$ref": "#/definitions/MyEnum"
#                 }
#             ]
#         }
#     },
#     "required": [
#         "my_field"
#     ],
#     "definitions": {
#         "MyEnum": {
#             "title": "MyEnum",
#             "description": "An enumeration.",
#             "enum": [
#                 "Foo",
#                 "Bar"
#             ],
#             "type": "string"
#         }
#     }

llm = ChatVertexAI(model_name='gemini-pro')
structured_llm = llm.with_structured_output(MyOutput)
# Key 'allOf' is not supported in schema, ignoring

structured_llm
# RunnableBinding(bound=ChatVertexAI(project='#####', model_name='gemini-pro', model_family=<GoogleModelFamily.GEMINI: '1'>, full_model_name='######/publishers/google/models/gemini-pro'), kwargs={'tools': [function_declarations {
#    name: "MyOutput"
#    parameters {
#      type_: OBJECT
#      title: "MyOutput"
#      properties {
#        key: "my_field"
#        value {
#          description: "This is my field"
#        }
#      }
#      required: "my_field"
#    }
#  }
#  ], 'tool_config': None})
#  | PydanticToolsParser(first_tool_only=True, tools=[<class '__main__.MyOutput'>])

structured_llm.invoke('Would you foo rather than bar?')
# Retrying langchain_google_vertexai.chat_models._completion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Unable to submit request because one or more function parameters didn't specify the schema type field. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling.

Expected result: the tool definitions should parse allOf properly Actual result: allOf is ignored and the result is a broken schema that fails to be invoked.

With ChatOpenAI this works as expected:

from langchain_openai import ChatOpenAI
gpt = ChatOpenAI(model="gpt-4o")
structured_gpt = gpt.with_structured_output(MyOutput)
structured_gpt
# RunnableBinding(bound=ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x11857fda0>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x1185b5010>, model_name='gpt-4o', openai_api_key=SecretStr('**********'), openai_proxy=''), kwargs={'tools': [{'type': 'function', 'function': {'name': 'MyOutput', 'description': '', 'parameters': {'type': 'object', 'properties': {'my_field': {'description': 'This is my field', 'allOf': [{'title': 'MyEnum', 'description': 'An enumeration.', 'enum': ['Foo', 'Bar'], 'type': 'string'}]}}, 'required': ['my_field']}}}], 'tool_choice': {'type': 'function', 'function': {'name': 'MyOutput'}}})
| PydanticToolsParser(first_tool_only=True, tools=[<class '__main__.MyOutput'>])

structured_gpt.invoke('Would you rather foo than bar?')
# MyOutput(my_field=<MyEnum.foo: 'Foo'>)

[hiravebapu]

any updates on this issue ? I am facing similar issue

[emersoftware]

add a type in the Field works for me:

class MyOutput(BaseModel):
    my_field: MyEnum = Field(...,description="This is my field", type="string")

[antti-ngp]

add a type in the Field works for me:

class MyOutput(BaseModel):
    my_field: MyEnum = Field(...,description="This is my field", type="string")

Did you actually try this? At least with

langchain==0.2.1
langchain-core==0.2.2
langchain-google-vertexai==1.0.5

this

class MyOutput(BaseModel):
    my_field: MyEnum = Field(...,description="This is my field", type="string")

results in the exact same error.

The problem seems to be the Field() definition as it results in the allOf key being added to the json schema. With just:

class MyOutput(BaseModel):
    my_field: MyEnum

it works, but then I need some other way to instruct the LLM what to do with this field.

[emersoftware]

add a type in the Field works for me:

class MyOutput(BaseModel):
    my_field: MyEnum = Field(...,description="This is my field", type="string")

Did you actually try this? At least with

langchain==0.2.1
langchain-core==0.2.2
langchain-google-vertexai==1.0.5

this

class MyOutput(BaseModel):
    my_field: MyEnum = Field(...,description="This is my field", type="string")

results in the exact same error.

The problem seems to be the Field() definition as it results in the allOf key being added to the json schema. With just:

class MyOutput(BaseModel):
    my_field: MyEnum

it works, but then I need some other way to instruct the LLM what to do with this field.

Oh, I see. Currently I am having the same issue.