Response schema seems having a length limit

[seungduk-yanolja]

Description of the bug:

I have been debugging a weird issue for the past few nights and finally found the tipping point between inputs that cause errors and those that do not.

I get an error if the schema is too long, and no error if the schema is shortened a little.

Actual vs expected behavior:

First, please note that I have converted the OpenAI schema into a Gemini one.

This version gives me an error,

{
  "temperature": 0,
  "model": "gemini-1.5-flash",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "properties": {
          "accuracy_of_message_delivery": {
            "$ref": "#/$defs/AppropriatenessOfLanguageUse"
          },
          "appropriateness_of_language_use": {
            "$ref": "#/$defs/AppropriatenessOfLanguageUse"
          }
        },
        "$defs": {
          "AppropriatenessOfLanguageUse": {
            "properties": {
              "verb_tenses_correct": {
                "$ref": "#/$defs/Score"
              },
              "conjunctions_and_transitions_effective": {
                "$ref": "#/$defs/Score"
              },
              "punctuation_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "pronouns_used_correctly": {
                "$ref": "#/$defs/Score"
              },
              "articles_and_prepositions_correct": {
                "$ref": "#/$defs/Score"
              },
              "capitalization_rules_followed": {
                "$ref": "#/$defs/Score"
              },
              "idiomatic_expressions_natural": {
                "$ref": "#/$defs/Score"
              },
              "vocabulary_choice_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "sentence_structures_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "subject_verb_agreement_maintained": {
                "$ref": "#/$defs/Score"
              },
              "active_passive_voice_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "grammar_rules_followed": {
                "$ref": "#/$defs/Score"
              },
              "technical_terminology_consistent": {
                "$ref": "#/$defs/Score"
              },
              "spelling_correct": {
                "$ref": "#/$defs/Score"
              }
            },
            "type": "object"
          },
          "Score": {
            "type": "object",
            "title": "Score",
            "properties": {
              "comment": {
                "type": "string",
                "title": "Comment"
              },
              "score": {
                "type": "integer",
                "title": "Score"
              }
            }
          }
        },
        "type": "object"
      }
    }
  },
  "messages": [
    {
      "content": "Evaluate the texts in a JSON format.",
      "role": "user"
    }
  ]
}

but this one does not.

{
  "temperature": 0,
  "model": "studio/gemini-1.5-flash",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "properties": {
          "accuracy_of_message_delivery": {
            "$ref": "#/$defs/AppropriatenessOfLanguageUse"
          },
          "appropriateness_of_language_use": {
            "$ref": "#/$defs/AppropriatenessOfLanguageUse"
          }
        },
        "$defs": {
          "AppropriatenessOfLanguageUse": {
            "properties": {
              "verb_tenses_correct": {
                "$ref": "#/$defs/Score"
              },
              "conjunctions_and_transitions_effective": {
                "$ref": "#/$defs/Score"
              },
              "punctuation_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "pronouns_used_correctly": {
                "$ref": "#/$defs/Score"
              },
              "articles_and_prepositions_correct": {
                "$ref": "#/$defs/Score"
              },
              "capitalization_rules_followed": {
                "$ref": "#/$defs/Score"
              },
              "idiomatic_expressions_natural": {
                "$ref": "#/$defs/Score"
              },
              "vocabulary_choice_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "sentence_structures_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "subject_verb_agreement_maintained": {
                "$ref": "#/$defs/Score"
              },
              "active_passive_voice_appropriate": {
                "$ref": "#/$defs/Score"
              },
              "grammar_rules_followed": {
                "$ref": "#/$defs/Score"
              },
              "technical_terminology_consistent": {
                "$ref": "#/$defs/Score"
              }
            },
            "type": "object"
          },
          "Score": {
            "type": "object",
            "title": "Score",
            "properties": {
              "comment": {
                "type": "string",
                "title": "Comment"
              },
              "score": {
                "type": "integer",
                "title": "Score"
              }
            }
          }
        },
        "type": "object"
      }
    }
  },
  "messages": [
    {
      "content": "Evaluate the texts in a JSON format.",
      "role": "user"
    }
  ]
}

Any other information you'd like to share?

The Go lang SDK does not support $defs and $ref so I converted them to the SDK supported format.

[seungduk-yanolja]

There is another issue also. https://github.com/google-gemini/generative-ai-python/issues/533

[eliben]

It sounds like an underlying API issue, not something specific to the Go SDK

It's worth asking on https://discuss.ai.google.dev/

[gmKeshari]

Hi @seungduk-yanolja. I found a similar issue on the Developers forum :

https://discuss.ai.google.dev/t/json-mode-internal-server-error-500/38123

I will escalate this.

Thanks!

[seungduk-yanolja]

Hi @seungduk-yanolja. I found a similar issue on the Developers forum :

https://discuss.ai.google.dev/t/json-mode-internal-server-error-500/38123

I will escalate this.

Thanks!

@gmKeshari Thanks! Please escalate this too: https://github.com/google-gemini/generative-ai-python/issues/533