[Question]: Adding top_p and topk parameters to gemini in vertex ai

[Prem-Nitin]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

Currently, i am using gemini from vertex ai for my rag and i am unable to change the top_p and top_k parameters of the llm. When i pass it in additional_kwargs, i just get errors. What to do? How to pass these parameters to the model?

[dosubot[bot]]

To correctly pass the top_p and top_k parameters to the Gemini model from Vertex AI for your RAG (Retrieval-Augmented Generation) without encountering errors, you should ensure that these parameters are included in the model's initialization or when calling the complete method. Here is an example of how to do this:

Initialization Example

vllm_instance = Vllm(
    model="your-model-name",
    temperature=1.0,
    top_p=0.9,  # Set your desired top_p value
    top_k=50,   # Set your desired top_k value
    # other parameters...
)

Method Call Example

response = vllm_instance.complete(
    prompt="Your prompt here",
    top_p=0.9,  # Set your desired top_p value
    top_k=50    # Set your desired top_k value
)

These parameters are part of the Vllm class and can be set either during initialization or when calling the complete method. The top_p parameter controls the cumulative probability of the top tokens to consider, and the top_k parameter controls the number of top tokens to consider ^[1].

To continue talking to Dosu, mention @dosu.