google-gemini / generative-ai-python

The official Python library for the Google Gemini API
https://pypi.org/project/google-generativeai/
Apache License 2.0
1.62k stars 322 forks source link

Unexpected behavior with some Spanish prompts than finish reason is OTHER #593

Open mdarino opened 1 month ago

mdarino commented 1 month ago

Description of the bug:

I have a Python script that generates menus by reading them from a list and completing a prompt (the prompts are in Spanish). However, for some reason, certain prompts return a ValueError. After some testing, I was able to reduce the issue to a small script, shown below:

import google.generativeai as genai
import os

genai.configure(api_key=os.environ["GEMINI_API_KEY"])

model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(
    'Genera una lista de ingredientes del siguiente menu: "Cafe con leche"',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
print(response)
print(response.text)

When I change the menu (e.g., to "Té con leche"), it fails, and the error message is not clear enough for me to understand the cause, it shows "finish_reason": "OTHER"

Actual vs expected behavior:

Some examples:

1 -

import google.generativeai as genai
import os

genai.configure(api_key=os.environ["GEMINI_API_KEY"])

model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(
    'Genera una lista de ingredientes del siguiente menu: "Cafe con leche"',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
print(response)
print(response.text)

RESULT OK:

response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "content": {
            "parts": [
              {
                "text": "## Ingredientes para Caf\u00e9 con Leche:\n\n* **Caf\u00e9:** (molido o en grano)\n* **Leche:** (entera, semidesnatada, desnatada, vegetal)\n* **Agua:** (para preparar el caf\u00e9)\n* **Az\u00facar:** (opcional)\n"
              }
            ],
            "role": "model"
          },
          "finish_reason": "STOP",
          "index": 0,
          "safety_ratings": [
            {
              "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HATE_SPEECH",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HARASSMENT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
              "probability": "NEGLIGIBLE"
            }
          ]
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 16,
        "candidates_token_count": 60,
        "total_token_count": 76
      }
    }),
)
## Ingredientes para Café con Leche:

* **Café:** (molido o en grano)
* **Leche:** (entera, semidesnatada, desnatada, vegetal)
* **Agua:** (para preparar el café)
* **Azúcar:** (opcional)

2 -

import google.generativeai as genai
import os

genai.configure(api_key=os.environ["GEMINI_API_KEY"])

model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(
    'Genera una lista de ingredientes del siguiente menu: "Te con leche"',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
print(response)
print(response.text)
response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "finish_reason": "OTHER",
          "index": 0
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 16,
        "total_token_count": 16
      }
    }),
)
Traceback (most recent call last):
  File "/home/mdarino/Desktop/foodcards/ai_menus/error.py", line 16, in <module>
    print(response.text)
          ^^^^^^^^^^^^^
  File "/home/mdarino/Desktop/foodcards/ai_menus/.venv/lib/python3.12/site-packages/google/generativeai/types/generation_types.py", line 495, in text
    raise ValueError(msg)
ValueError: Invalid operation: The `response.text` quick accessor requires the response to contain a valid `Part`, but none were returned. The candidate's [finish_reason](https://ai.google.dev/api/generate-content#finishreason) is 5.

Any other information you'd like to share?

Another example:

Prompt: 'Genera una lista de ingredientes del siguiente menu: "Pizza Mexicana (carne, jalapeños, frijoles, mozzarella)"',

RESULT NOT OK:

response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "finish_reason": "OTHER",
          "index": 0
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 27,
        "total_token_count": 27
      }
    }),
)
Traceback (most recent call last):
  File "/home/mdarino/Desktop/foodcards/ai_menus/error.py", line 16, in <module>
    print(response.text)
          ^^^^^^^^^^^^^
  File "/home/mdarino/Desktop/foodcards/ai_menus/.venv/lib/python3.12/site-packages/google/generativeai/types/generation_types.py", line 495, in text
    raise ValueError(msg)
ValueError: Invalid operation: The `response.text` quick accessor requires the response to contain a valid `Part`, but none were returned. The candidate's [finish_reason](https://ai.google.dev/api/generate-content#finishreason) is 5.

But Prompt: 'Genera una lista de ingredientes del siguiente menu: "Pizza de camarones (camarones, ajo, mozzarella)"',

RETURN OK:

response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "content": {
            "parts": [
              {
                "text": "## Ingredientes para Pizza de Camarones:\n\n* **Base de pizza:** (Harina, agua, levadura, aceite de oliva, sal)\n* **Salsa de tomate:** (Tomates, cebolla, ajo, especias)\n* **Camarones:** (Frescos o congelados)\n* **Ajo:** (Picado o en polvo)\n* **Mozzarella:** (Rallada o en rebanadas)\n* **Aceite de oliva:** (Para rociar)\n* **Or\u00e9gano:** (Para espolvorear)\n* **Sal y pimienta:** (Al gusto)\n\n**Opcionales:**\n\n* **Ceboll\u00edn:** (Picado para decorar)\n* **Perejil:** (Picado para decorar)\n* **Queso parmesano:** (Rallado para espolvorear)\n* **Piment\u00f3n dulce:** (Para espolvorear)\n* **Aj\u00ed picante:** (Para agregar un toque picante)\n"
              }
            ],
            "role": "model"
          },
          "finish_reason": "STOP",
          "index": 0,
          "safety_ratings": [
            {
              "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HATE_SPEECH",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HARASSMENT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
              "probability": "NEGLIGIBLE"
            }
          ]
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 25,
        "candidates_token_count": 205,
        "total_token_count": 230
      }
    }),
)
## Ingredientes para Pizza de Camarones:

* **Base de pizza:** (Harina, agua, levadura, aceite de oliva, sal)
* **Salsa de tomate:** (Tomates, cebolla, ajo, especias)
* **Camarones:** (Frescos o congelados)
* **Ajo:** (Picado o en polvo)
* **Mozzarella:** (Rallada o en rebanadas)
* **Aceite de oliva:** (Para rociar)
* **Orégano:** (Para espolvorear)
* **Sal y pimienta:** (Al gusto)

**Opcionales:**

* **Cebollín:** (Picado para decorar)
* **Perejil:** (Picado para decorar)
* **Queso parmesano:** (Rallado para espolvorear)
* **Pimentón dulce:** (Para espolvorear)
* **Ají picante:** (Para agregar un toque picante)
mdarino commented 1 month ago

In English: 'Generate a list of ingredients for the following menu: "Te con leche"' Work well On www.gemini.google.com, all prompts respond as expected (just in case I tested them)

gmKeshari commented 1 month ago

Hi @mdarino

I have escalated this to internal team.