top_p=0 Parameter Not Functioning as Expected in Vertex Gemini AI

[storybite]

The top_p=0 parameter, intended to produce the most probable and consistent response, does not work as expected in Vertex Gemini AI. Unlike its intended functionality, which is to limit output variability for identical inputs, Vertex Gemini AI exhibits significant response variability under this setting.

In comparison, Google Gemini AI demonstrates the expected behavior with top_p=0, providing nearly identical responses for repeated inputs.

import google.generativeai as genai
from vertexai.preview.generative_models import GenerativeModel, GenerationConfig 

v_model = GenerativeModel(model_name='gemini-pro')    
v_generation_config = GenerationConfig(top_p=0)

g_model = genai.GenerativeModel(model_name='gemini-pro')
g_generation_config = genai.GenerationConfig(top_p=0)

user_message = "Write a one-sentence poem"

print("\nVertex Gemini AI top_p=0:")
for _ in range(3):
    response = v_model.generate_content(user_message , generation_config=v_generation_config)
    print(f'{"_"*20}{'\n'}{response.text}')    

print("\nGoogle Gemini AI top_p=0:")
for _ in range(3):
    response = g_model.generate_content(user_message , generation_config=g_generation_config)
    print(f'{"_"*20}{'\n'}{response.text}')

output:

Vertex Gemini AI top_p=0:
____________________
In the realm of shadows, a heart beats alone.
____________________
"A flicker of light, a fleeting dream."
____________________
In golden light, a solitary crow takes flight.

Google Gemini AI top_p=0:
____________________
In the vast expanse, a star whispers its tale.
____________________
In the vast expanse, a star whispers its tale.
____________________
In the vast expanse, a star whispers its tale.

I think Vertex Gemini AI should also work like Google Gemini AI.

Environment details:

Python version: 3.12.1 pip version: 23.2.1 google-cloud-aiplatform version: 1.40.0

[ZhenyiQ]

Hi, it is determined that this is an intended behavior for now since the underlying model may also change (therefore there is currently no way to keep the output deterministic). Will mark this closed.