krrishdholakia
commented
11 months ago User needs it to support sending top_k, top_p, etc. parameters to their vllm model
Open krrishdholakia opened 11 months ago
krrishdholakia
commented
11 months ago User needs it to support sending top_k, top_p, etc. parameters to their vllm model
The Feature
The normal vllm server, supports these inputs. By supporting this, we can handle prompt formatting on the proxy instead of the individual vllm server
Motivation, pitch
User had to do more setup work once their vllm - litellm setup was done, to support the individual prompt template
Twitter / LinkedIn details
No response