`gemini-1.5-flash` does not generate JSON according to provided Schema.

[PCloud-Bernard]

Description of the bug:

While using gemini-1.5-flash, no matter how I define the response schema, it does not respect the defined schema but gemini-1.5-pro-latest does not have this issue.

Code snippet:

ctx := context.Background()
client, err := genai.NewClient(ctx, option.WithAPIKey(os.Getenv("GEMINI_API_KEY")))
if err != nil {
    log.Fatal(err)
}
defer client.Close()

model := client.GenerativeModel("gemini-1.5-flash")
// Ask the model to respond with JSON.
model.ResponseMIMEType = "application/json"
// Specify the format of the JSON.
model.ResponseSchema = &genai.Schema{
    Type: genai.TypeArray,
    Items: &genai.Schema{
        Type: genai.TypeObject,
        Properties: map[string]*genai.Schema{
            "name": {
                Type:        genai.TypeString,
                Description: "The name of the color",
            },
            "RGB": {
                Type:        genai.TypeString,
                Description: "The RGB value of the color, in hex",
            },
        },
        Required: []string{"name", "RGB"},
    },
}

resp, err := model.GenerateContent(ctx, genai.Text("List the primary colors."))

if err != nil {
    log.Error("Generate Content Error: ", err)
    return ""
}

if (len(resp.Candidates) > 0) {
    content := resp.Candidates[0].Content

    if (content != nil) {
        if (len(content.Parts) > 0) {
            part := content.Parts[0]

            log.Println(fmt.Sprintf("%s", part))
        }
    }
}

Actual vs expected behavior:

Actual Response:

{
  "primary_colors": [
    "red",
    "yellow",
    "blue"
  ]
}

Expected Response:

[
  {
    "RGB": "255, 0, 0",
    "name": "Red"
  },
  {
    "RGB": "0, 255, 0",
    "name": "Green"
  },
  {
    "RGB": "0, 0, 255",
    "name": "Blue"
  }
]

Any other information you'd like to share?

No response

[eliben]

If the only thing you're changing is the model name, this is likely to be an issue with the underlying service or model, not the Go SDK. Please consider asking on the general support forum https://discuss.ai.google.dev/ -- where folks who use other languages could also have encountered this issue

[PCloud-Bernard]

After asking on the general support forum, I've gotten the answer. I might as well update here for people who want to know as well.

gemini-1.5-flash only accepts JSON schema from text, as in prompt or system instruction, whereas gemini-1.5-pro takes schema object.

The documentation and cookbook shows how to define schemas in different models.