search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.65k stars 1.47k forks source link

[Feature]: Support `seed` for Gemini in Vertex AI #2973

Closed Manouchehri closed 2 months ago

Manouchehri commented 5 months ago

The Feature

While not documented, seed is "supported" now. https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#parameter-list

Similar to #2962. Related to https://github.com/googleapis/python-aiplatform/pull/3186.

Motivation, pitch

While seed doesn't currently give properly reproducible outputs, it does seem to help. (And it'll probably/hopefully work better once it's officially supported.)

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

Manouchehri commented 5 months ago

It's been hidden from the docs now, but it's still accepted by the API:

curl -X POST "https://us-east4-aiplatform.googleapis.com/v1/projects/litellm-epic/locations/us-east4/publishers/google/models/gemini-1.5-pro-preview-0409:generateContent" \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H 'Content-Type: application/json' \
    -d '{
    "contents": [
        {
            "role": "user",
            "parts": [
                {
                    "text": "tell me a joke"
                }
            ]
        }
    ],
    "safety_settings": [
      {
        "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
        "threshold": "BLOCK_LOW_AND_ABOVE"
      }
    ],
    "generation_config": {
      "max_output_tokens": 256,
      "response_mime_type": "application/json",
      "seed": 1337
    }
}' -v
Manouchehri commented 5 months ago

Related, https://github.com/googleapis/python-aiplatform/pull/3639 should remove the need for the hacky current solution.

Manouchehri commented 5 months ago

I'll wait for the HTTP client refactor of Vertex AI before doing this.

IA-Programming commented 4 months ago

So still there's no support for function calling with gemini-1.5 vertex model?

Manouchehri commented 2 months ago

@IA-Programming It has been added in #4588.