search

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)
https://docs.litellm.ai/docs/
Other
10.38k stars 1.17k forks source link

[Feature]: Support TGI 'truncate' param #1739

Closed krrishdholakia closed 5 months ago

krrishdholakia commented 5 months ago

The Feature

TGI supports a truncate param for handling scenarios where max tokens > model limit

support it

Motivation, pitch

user request

Twitter / LinkedIn details

No response

krrishdholakia commented 5 months ago

this wasn't working - https://github.com/huggingface/chat-ui/blob/main/.env#L75

krrishdholakia commented 5 months ago

no-op our optional param handling seems to work fine image