search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
13.63k stars 1.6k forks source link

[Bug]: Proxy config - model specific drop_params doesn't work #5501

Closed kresimirfijacko closed 2 months ago

kresimirfijacko commented 2 months ago

What happened?

Version: ghcr.io/berriai/litellm:main-v1.44.14

I have problem setting 'drop_params' on specific model, which should be supported by documentation https://docs.litellm.ai/docs/completion/drop_params

example proxy_config file: ` model_list:

litellm_settings: drop_params: True # THIS WORKS

`

Relevant log output

No response

Twitter / LinkedIn details

No response

srail commented 2 months ago

On a potentially related note, seeding a seed parameter for vertex/llama 3.1 405 errors out when with drop_params set to true. Same parameter works for azure / gpt-4 calls.

krrishdholakia commented 2 months ago

missed this @kresimirfijacko, i believe drop param support for embedding was added in a recent version - can you confirm this persists on latest?

krrishdholakia commented 2 months ago

Hi @kresimirfijacko unable to repro, this works for me as expected.

Output

Screenshot 2024-09-06 at 9 50 51 AM

Curl

curl -L -X POST 'http://0.0.0.0:4000/embeddings' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{"input": ["hello world"], "model": "embedding", "dimensions": 3}'

Config

model_list:
  - model_name: embedding
    litellm_params:
      model: bedrock/amazon.titan-embed-text-v1
      drop_params: True
krrishdholakia commented 2 months ago

@srail vertex ai llama3.1 works with seed param (see below)

Screenshot 2024-09-06 at 10 05 28 AM

Vertex Llama3.1 is currently assumed to be completely openai compatible - https://github.com/BerriAI/litellm/blob/4626c5a365c725ed21292376bb1d9ba3a74fdfab/litellm/llms/vertex_ai_and_google_ai_studio/vertex_ai_partner_models/llama3/transformation.py#L52

as I can't find any docs on the complete list of vertex llama 3 param docs (e.g. their example on vertex shows max_tokens, but their playground has max_tokens, top_p, top_k and temperature)

ishaan-jaff commented 1 month ago

Hi @kresimirfijacko , curious do you use LiteLLM today ? If so, I'd love to hop on a call and learn how we can improve LiteLLM for you