BatchPrediction 400: Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema

orenmiaraswish commented 2 months ago

Problem:

400 Error message With Vertex BatchPrediction API with gemini-1.5-flash-001 and gemini-1.5-pro-001

I am getting similar issues using Batch Prediction with gemini-1.5-flash-001 and gemini-1.5-pro-001. When not using Vertex BatchPrediction API the code at the (Expected output) works.

followed the documentation here:

https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output#example_use_cases
https://cloud.google.com/vertex-ai/docs/reference/rest/v1/Schema
https://github.com/googleapis/python-aiplatform/blob/main/vertexai/generative_models/_generative_models.py#L1891
Environment details
Windows 10 Pro 22H2
python 3.11.8 / pip 24.2
google-cloud-aiplatform==1.64.0
google-cloud-bigquery==3.18.0
google-cloud-core==2.4.1
google-cloud-kms==2.21.1
google-api-core==2.19.1
google-resumable-media==2.7.2
googleapis-common-protos==1.63.2
Google Cloud SDK 482.0.0 (beta 2024.06.24, core 2024.06.24, gcloud-crc32c 1.0.0, gke-gcloud-auth-plugin 0.5.8 gsutil 5.30)
LOCATION : europe-west4

Steps to reproduce:

Use Vertex BatchPrediction API (europe-west4)
Use BigQuery

Run the Job with Input payload as below (Taken from BigQuery Input):

{
"contents":
[
    {
        "role": "user",
        "parts":
        {
            "text": "TEXT GOES HERE"
        }
    }
],
"system_instruction":
{
    "parts":
    [
        {
            "text": "MY SYSTEM PROMPT"
        }
    ]
},
"safety_settings":
[
    {
        "category": 1,
        "threshold": 4
    },
    {
        "category": 3,
        "threshold": 4
    },
    {
        "category": 4,
        "threshold": 4
    },
    {
        "category": 2,
        "threshold": 4
    },
    {
        "category": 0,
        "threshold": 4
    }
],
"generation_config":
{
    "temperature": 0.1,
    "response_mime_type": "application/json",
    "response_schema":
    {
        "type_": "OBJECT",
        "properties":
        {
            "description":
            {
                "type_": "STRING"
            },
            "steps":
            {
                "type_": "BOOLEAN"
            },
            "title":
            {
                "type_": "STRING"
            },
            "questions":
            {
                "type_": "ARRAY",
                "items":
                {
                    "type_": "STRING"
                }
            }
        },
        "required":
        [
            "title",
            "steps",
            "description",
            "questions"
        ]
    }
}
}

Code example

generation_config = GenerationConfig(
    temperature=0.1,
    response_mime_type="application/json",
    response_schema={
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "steps": {"type": "boolean"},
            "description": {"type": "string"},
            "questions": {"type": "array", "items": {"type": "string"}}
        },
        "required": [
            "title",
            "steps",
            "description",
            "questions"
        ]
    }
)
...

Stack trace: (BigQuery - Output)

{
    "error":
    {
        "code": 400,
        "message": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[0].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[1].value.items': Cannot find field.\nInvalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[1].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[2].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[3].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema': Cannot find field.",
        "status": "INVALID_ARGUMENT",
        "details":
        [
            {
                "@type": "type.googleapis.com/google.rpc.BadRequest",
                "fieldViolations":
                [
                    {
                        "field": "generation_config.response_schema.properties[0].value",
                        "description": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[0].value': Cannot find field."
                    },
                    {
                        "field": "generation_config.response_schema.properties[1].value.items",
                        "description": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[1].value.items': Cannot find field."
                    },
                    {
                        "field": "generation_config.response_schema.properties[1].value",
                        "description": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[1].value': Cannot find field."
                    },
                    {
                        "field": "generation_config.response_schema.properties[2].value",
                        "description": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[2].value': Cannot find field."
                    },
                    {
                        "field": "generation_config.response_schema.properties[3].value",
                        "description": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema.properties[3].value': Cannot find field."
                    },
                    {
                        "field": "generation_config.response_schema",
                        "description": "Invalid JSON payload received. Unknown name \"type_\" at 'generation_config.response_schema': Cannot find field."
                    }
                ]
            }
        ]
    }
}

Expected Results Example:

BigQuery Output table Response column filled with the adequate JSON output.

model = GenerativeModel(
    model_name='gemini-1.5-pro-001',
    generation_config=GenerationConfig(
        response_mime_type="application/json",
        response_schema={
            "type": "object",
            "properties": {
                "title": {"type": "STRING"},
                "steps": {"type": "BOOLEAN"},
                "description": {"type": "STRING"},
                "questions": {"type": "ARRAY", "items": {"type": "STRING"}}
            },
            "required": [
                "title",
                "steps",
                "description",
                "questions",
            ],
        }
    )
)

response = await model.generate_content_async("Is this sentce possitive negative or neutral? I am angry and happier")
{
    "description": "This sentence expresses conflicting emotions, making it difficult to categorize as strictly positive, negative, or neutral. It suggests a mix of anger and happiness, which could be interpreted as a complex emotional state.",
    "questions":
    [
        "Is the sentence expressing a positive, negative, or neutral sentiment?"
    ],
    "steps": false,
    "title": "Analyzing Sentence Sentiment"
}

Ark-kun commented 2 months ago

@jaycee-li Can you check whether the schema proto is correctly serialized to JSON via json_format?

Or could this be caused by bugs in the code that handles proto fields renamed by the GAPIC generator?

jaycee-li commented 2 months ago

Hi @orenmiaraswish,

It seems like the issue lies in the format of your Bigquery input. Bigquery expects requests in JSON format, whereas your input is currently a raw proto message.

The key difference is that Schema proto uses fields type_ and format_, while its JSON counterpart uses type and format.

To resolve your issue, you'll need to modify your Bigquery input by changing type_ to type in response_shcema.

orenmiaraswish commented 2 months ago

@jaycee-li thanks for your replay.

The BigQuery Input is actually a JSON, though it was originated from GenerationConfig.to_dict() thus contains "type_"... I overcome this issue by using plain JSON, to not going through the gapic validation stuff.

Original Code:

...
@staticmethod
def raw_to_json(request_id, prompt, instructions, generation_config):
    if isinstance(generation_config, GenerationConfig):
        generation_config = generation_config.to_dict()
    request = {
        "contents": [{"role": "user", "parts": {"text": prompt}}],
        "system_instruction": {"parts": [{"text": f"{instructions}"}]},
        "safety_settings": batch_request_safety_settings,
        "generation_config": generation_config,
    }
    return {"id": request_id, "request": json.dumps(request)}
...

ItsOkayItsOfficial commented 1 month ago

I think we are getting a variant of the same response when using the Python SDK.

Using the Vertex AI Studio with Output format JSON enabled does produced the desired response.

The Get code section provides incorrect code (below).

However after correcting the proffered code our generation_config looks like this:

generation_config = {
    "max_output_tokens": 8192,
    "temperature": 0.5,
    "top_p": 0.95,
    "response_mime_type": "application/json",
    "response_schema": {"type":"OBJECT","properties":{"document_type":{"type":"STRING"},"carrier":{"type":"STRING"},"shipped_date":{"type":"STRING"},"received_date":{"type":"STRING"},"purchase_order":{"type":"NUMBER"},"ship_from_complete_address":{"type":"STRING"},"ship_from_street_address":{"type":"STRING"},"ship_from_city":{"type":"STRING"},"ship_from_state":{"type":"STRING"},"ship_from_zip":{"type":"STRING"},"ship_from_name":{"type":"STRING"},"ship_to_complete_address":{"type":"STRING"},"ship_to_street_address":{"type":"STRING"},"ship_to_city":{"type":"STRING"},"ship_to_state":{"type":"STRING"},"ship_to_zip":{"type":"STRING"},"ship_to_name":{"type":"STRING"},"supplier_address":{"type":"STRING"},"supplier_name":{"type":"STRING"},"total_weight_received":{"type":"NUMBER"},"total_cases_received":{"type":"NUMBER"},"total_cube_received":{"type":"NUMBER"}},"required":["document_type","carrier","shipped_date","received_date","purchase_order","ship_from_complete_address","ship_from_street_address","ship_from_city","ship_from_state","ship_from_zip","ship_from_name","ship_to_complete_address","ship_to_street_address","ship_to_city","ship_to_state","ship_to_zip","ship_to_name","supplier_address","supplier_name","total_weight_received","total_cases_received","total_cube_received"]}
}

This produces the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /usr/local/lib/python3.10/dist-packages/proto/message.py:729, in Message.__init__(self, mapping, ignore_unknown_fields, **kwargs)
    [728](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:728) try:
--> [729](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:729)     pb_value = marshal.to_proto(pb_type, value)
    [730](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:730) except ValueError:
    [731](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:731)     # Underscores may be appended to field names
    [732](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:732)     # that collide with python or proto-plus keywords.
   (...)
    [736](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:736)     # See related issue
    [737](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:737)     # https://github.com/googleapis/python-api-core/issues/227

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235, in BaseMarshal.to_proto(self, proto_type, value, strict)
    [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
--> [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
    [237](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:237) # Sanity check: If we are in strict mode, did we get the value we want?

File /usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36, in MessageRule.to_proto(self, value)
     [34](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:34) try:
     [35](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:35)     # Try the fast path first.
---> [36](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36)     return self._descriptor(**value)
     [37](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:37) except TypeError as ex:
     [38](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:38)     # If we have a type error,
     [39](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:39)     # try the slow path in case the error
     [40](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:40)     # was an int64/string issue

ValueError: Protocol message Schema has no "type" field.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
File /usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36, in MessageRule.to_proto(self, value)
     [34](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:34) try:
     [35](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:35)     # Try the fast path first.
---> [36](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36)     return self._descriptor(**value)
     [37](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:37) except TypeError as ex:
     [38](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:38)     # If we have a type error,
     [39](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:39)     # try the slow path in case the error
     [40](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:40)     # was an int64/string issue

TypeError: Parameter to CopyFrom() must be instance of same class: expected <class 'Schema'> got <class 'dict'>.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
File /usr/local/lib/python3.10/dist-packages/proto/message.py:729, in Message.__init__(self, mapping, ignore_unknown_fields, **kwargs)
    [728](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:728) try:
--> [729](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:729)     pb_value = marshal.to_proto(pb_type, value)
    [730](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:730) except ValueError:
    [731](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:731)     # Underscores may be appended to field names
    [732](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:732)     # that collide with python or proto-plus keywords.
   (...)
    [736](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:736)     # See related issue
    [737](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:737)     # https://github.com/googleapis/python-api-core/issues/227

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233, in BaseMarshal.to_proto(self, proto_type, value, strict)
    [232](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:232)     recursive_type = type(proto_type().value)
--> [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
    [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233, in <dictcomp>(.0)
    [232](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:232)     recursive_type = type(proto_type().value)
--> [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
    [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235, in BaseMarshal.to_proto(self, proto_type, value, strict)
    [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
--> [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
    [237](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:237) # Sanity check: If we are in strict mode, did we get the value we want?

File /usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36, in MessageRule.to_proto(self, value)
     [34](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:34) try:
     [35](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:35)     # Try the fast path first.
---> [36](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36)     return self._descriptor(**value)
     [37](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:37) except TypeError as ex:
     [38](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:38)     # If we have a type error,
     [39](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:39)     # try the slow path in case the error
     [40](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:40)     # was an int64/string issue

ValueError: Protocol message Schema has no "type" field.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[2], [line 63](vscode-notebook-cell:?execution_count=2&line=63)
     [36](vscode-notebook-cell:?execution_count=2&line=36) generation_config = {
     [37](vscode-notebook-cell:?execution_count=2&line=37)     "max_output_tokens": 8192,
     [38](vscode-notebook-cell:?execution_count=2&line=38)     "temperature": 0.5,
   (...)
     [41](vscode-notebook-cell:?execution_count=2&line=41)     "response_schema": {"type":"OBJECT","properties":{"document_type":{"type":"STRING"},"carrier":{"type":"STRING"},"shipped_date":{"type":"STRING"},"received_date":{"type":"STRING"},"purchase_order":{"type":"NUMBER"},"ship_from_complete_address":{"type":"STRING"},"ship_from_street_address":{"type":"STRING"},"ship_from_city":{"type":"STRING"},"ship_from_state":{"type":"STRING"},"ship_from_zip":{"type":"STRING"},"ship_from_name":{"type":"STRING"},"ship_to_complete_address":{"type":"STRING"},"ship_to_street_address":{"type":"STRING"},"ship_to_city":{"type":"STRING"},"ship_to_state":{"type":"STRING"},"ship_to_zip":{"type":"STRING"},"ship_to_name":{"type":"STRING"},"supplier_address":{"type":"STRING"},"supplier_name":{"type":"STRING"},"total_weight_received":{"type":"NUMBER"},"total_cases_received":{"type":"NUMBER"},"total_cube_received":{"type":"NUMBER"}},"required":["document_type","carrier","shipped_date","received_date","purchase_order","ship_from_complete_address","ship_from_street_address","ship_from_city","ship_from_state","ship_from_zip","ship_from_name","ship_to_complete_address","ship_to_street_address","ship_to_city","ship_to_state","ship_to_zip","ship_to_name","supplier_address","supplier_name","total_weight_received","total_cases_received","total_cube_received"]}
     [42](vscode-notebook-cell:?execution_count=2&line=42) }
     [44](vscode-notebook-cell:?execution_count=2&line=44) safety_settings = [
     [45](vscode-notebook-cell:?execution_count=2&line=45)     SafetySetting(
     [46](vscode-notebook-cell:?execution_count=2&line=46)         category=SafetySetting.HarmCategory.HARM_CATEGORY_HATE_SPEECH,
   (...)
     [60](vscode-notebook-cell:?execution_count=2&line=60)     )
     [61](vscode-notebook-cell:?execution_count=2&line=61) ]
---> [63](vscode-notebook-cell:?execution_count=2&line=63) generate()

Cell In[2], [line 20](vscode-notebook-cell:?execution_count=2&line=20)
      [9](vscode-notebook-cell:?execution_count=2&line=9) model = GenerativeModel(
     [10](vscode-notebook-cell:?execution_count=2&line=10)   "gemini-1.5-flash-001",
     [11](vscode-notebook-cell:?execution_count=2&line=11)   system_instruction=[textsi_1]
     [12](vscode-notebook-cell:?execution_count=2&line=12) )
     [13](vscode-notebook-cell:?execution_count=2&line=13) responses = model.generate_content(
     [14](vscode-notebook-cell:?execution_count=2&line=14)     ["""Extract the text value of the entities in this document containing a Bill of Lading.""", document1],
     [15](vscode-notebook-cell:?execution_count=2&line=15)     generation_config=generation_config,
     [16](vscode-notebook-cell:?execution_count=2&line=16)     safety_settings=safety_settings,
     [17](vscode-notebook-cell:?execution_count=2&line=17)     stream=True,
     [18](vscode-notebook-cell:?execution_count=2&line=18) )
---> [20](vscode-notebook-cell:?execution_count=2&line=20) for response in responses:
     [21](vscode-notebook-cell:?execution_count=2&line=21)   print(response.text, end="")

File ~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:769, in _GenerativeModel._generate_content_streaming(self, contents, generation_config, safety_settings, tools, tool_config)
    [742](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:742) def _generate_content_streaming(
    [743](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:743)     self,
    [744](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:744)     contents: ContentsType,
   (...)
    [749](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:749)     tool_config: Optional["ToolConfig"] = None,
    [750](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:750) ) -> Iterable["GenerationResponse"]:
    [751](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:751)     """Generates content.
    [752](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:752) 
    [753](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:753)     Args:
   (...)
    [767](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:767)         A stream of GenerationResponse objects
    [768](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:768)     """
--> [769](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:769)     request = self._prepare_request(
    [770](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:770)         contents=contents,
    [771](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:771)         generation_config=generation_config,
    [772](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:772)         safety_settings=safety_settings,
    [773](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:773)         tools=tools,
    [774](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:774)         tool_config=tool_config,
    [775](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:775)     )
    [776](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:776)     response_stream = self._prediction_client.stream_generate_content(
    [777](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:777)         request=request
    [778](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:778)     )
    [779](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:779)     for chunk in response_stream:

File ~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:467, in _GenerativeModel._prepare_request(self, contents, generation_config, safety_settings, tools, tool_config, system_instruction)
    [465](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:465)         gapic_generation_config = generation_config._raw_generation_config
    [466](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:466)     elif isinstance(generation_config, Dict):
--> [467](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:467)         gapic_generation_config = gapic_content_types.GenerationConfig(
    [468](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:468)             **generation_config
    [469](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:469)         )
    [471](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:471) gapic_safety_settings = None
    [472](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/home/username/genai/generate_response/~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:472) if safety_settings:

File /usr/local/lib/python3.10/dist-packages/proto/message.py:757, in Message.__init__(self, mapping, ignore_unknown_fields, **kwargs)
    [754](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:754)         for item in keys_to_update:
    [755](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:755)             value[f"{item}_"] = value.pop(item)
--> [757](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:757)     pb_value = marshal.to_proto(pb_type, value)
    [759](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:759) if pb_value is not None:
    [760](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:760)     params[key] = pb_value

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235, in BaseMarshal.to_proto(self, proto_type, value, strict)
    [232](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:232)     recursive_type = type(proto_type().value)
    [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
--> [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
    [237](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:237) # Sanity check: If we are in strict mode, did we get the value we want?
    [238](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:238) if strict and not isinstance(pb_value, proto_type):

File /usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:41, in MessageRule.to_proto(self, value)
     [36](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36)         return self._descriptor(**value)
     [37](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:37)     except TypeError as ex:
     [38](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:38)         # If we have a type error,
     [39](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:39)         # try the slow path in case the error
     [40](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:40)         # was an int64/string issue
---> [41](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:41)         return self._wrapper(value)._pb
     [42](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:42) return value

File /usr/local/lib/python3.10/dist-packages/proto/message.py:757, in Message.__init__(self, mapping, ignore_unknown_fields, **kwargs)
    [754](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:754)         for item in keys_to_update:
    [755](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:755)             value[f"{item}_"] = value.pop(item)
--> [757](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:757)     pb_value = marshal.to_proto(pb_type, value)
    [759](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:759) if pb_value is not None:
    [760](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/message.py:760)     params[key] = pb_value

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233, in BaseMarshal.to_proto(self, proto_type, value, strict)
    [228](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:228) if isinstance(value, dict) and (
    [229](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:229)     proto_type.DESCRIPTOR.has_options
    [230](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:230)     and proto_type.DESCRIPTOR.GetOptions().map_entry
    [231](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:231) ):
    [232](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:232)     recursive_type = type(proto_type().value)
--> [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
    [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
    [237](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:237) # Sanity check: If we are in strict mode, did we get the value we want?

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233, in <dictcomp>(.0)
    [228](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:228) if isinstance(value, dict) and (
    [229](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:229)     proto_type.DESCRIPTOR.has_options
    [230](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:230)     and proto_type.DESCRIPTOR.GetOptions().map_entry
    [231](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:231) ):
    [232](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:232)     recursive_type = type(proto_type().value)
--> [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
    [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
    [237](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:237) # Sanity check: If we are in strict mode, did we get the value we want?

File /usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235, in BaseMarshal.to_proto(self, proto_type, value, strict)
    [232](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:232)     recursive_type = type(proto_type().value)
    [233](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:233)     return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
--> [235](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:235) pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
    [237](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:237) # Sanity check: If we are in strict mode, did we get the value we want?
    [238](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/marshal.py:238) if strict and not isinstance(pb_value, proto_type):

File /usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36, in MessageRule.to_proto(self, value)
     [31](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:31) if isinstance(value, dict) and not self.is_map:
     [32](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:32)     # We need to use the wrapper's marshaling to handle
     [33](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:33)     # potentially problematic nested messages.
     [34](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:34)     try:
     [35](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:35)         # Try the fast path first.
---> [36](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:36)         return self._descriptor(**value)
     [37](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:37)     except TypeError as ex:
     [38](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:38)         # If we have a type error,
     [39](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:39)         # try the slow path in case the error
     [40](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:40)         # was an int64/string issue
     [41](https://vscode-remote+970-002dcs-002d233782833769-002ddefault-002ecs-002dus-002dwest1-002dwolo-002ecloudshell-002edev.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/dist-packages/proto/marshal/rules/message.py:41)         return self._wrapper(value)._pb

ValueError: Protocol message Schema has no "type" field.

ItsOkayItsOfficial commented 1 month ago

Following up on our previous comment above

We were able to run this successfully by replacing all references to type with type_

This generation_config produces the expected outcomes.

generation_config = {
    "max_output_tokens": 8192,
    "temperature": 0.5,
    "top_p": 0.95,
    "response_mime_type": "application/json",
    "response_schema": {"type_":"OBJECT","properties":{"document_type_":{"type_":"STRING"},"carrier":{"type_":"STRING"},"shipped_date":{"type_":"STRING"},"received_date":{"type_":"STRING"},"purchase_order":{"type_":"NUMBER"},"ship_from_complete_address":{"type_":"STRING"},"ship_from_street_address":{"type_":"STRING"},"ship_from_city":{"type_":"STRING"},"ship_from_state":{"type_":"STRING"},"ship_from_zip":{"type_":"STRING"},"ship_from_name":{"type_":"STRING"},"ship_to_complete_address":{"type_":"STRING"},"ship_to_street_address":{"type_":"STRING"},"ship_to_city":{"type_":"STRING"},"ship_to_state":{"type_":"STRING"},"ship_to_zip":{"type_":"STRING"},"ship_to_name":{"type_":"STRING"},"supplier_address":{"type_":"STRING"},"supplier_name":{"type_":"STRING"},"total_weight_received":{"type_":"NUMBER"},"total_cases_received":{"type_":"NUMBER"},"total_cube_received":{"type_":"NUMBER"}},"required":["document_type_","carrier","shipped_date","received_date","purchase_order","ship_from_complete_address","ship_from_street_address","ship_from_city","ship_from_state","ship_from_zip","ship_from_name","ship_to_complete_address","ship_to_street_address","ship_to_city","ship_to_state","ship_to_zip","ship_to_name","supplier_address","supplier_name","total_weight_received","total_cases_received","total_cube_received"]}
}

jaycee-li commented 1 month ago

Hi @ItsOkayItsOfficial , thanks for raising this issue! This is a bug and I'm fixing it. Both type and type_ should work here.

schwanne commented 3 weeks ago

I had a very similar experience as previous comment but with format. Replacing all instances of format with format_ in the code copied from VertexAI Studio got it working.

googleapis / python-aiplatform