Open mrdjohnson opened 3 months ago
Hey @mrdjohnson can't you already do this via - https://docs.litellm.ai/docs/providers/openai_compatible
model_list:
- model_name: my-model
litellm_params:
model: openai/<your-model-name> # your lm studio model name
api_base: <model-api-base> # lm studio api base
api_key: api-key # lm-studio api key if required (else put a fake key here)
unclear on what first-party support here looks like?
Thats very true, LM Studio is quite popular but its lacking in the first-party support section!
Its awesome that there is openai api compatibility but there are things the SDK gives like better insight into model generation statistics, listing all the downloaded models, loading models and the progress around loading the model, and unloading models. These are things that the open ai api does not account for.
I second that. LM Studio is the most popular and most commonly used, probably next to Ollama for local models. It's not listed as supported with specific features related to it.
I'm working with the LM Studio team! There are a lot of things the Typescript SDK can do that the api does not provide just yet. As soon as we get the api to allow more options i'll be able to provide more reasons to first party support!
Hey @mrdjohnson thanks! happy to add native support once you think it's ready
Took me a few tries and unfriendly error messages to figure out that api_base
is supposed to be http://localhost:1234/v1
so sharing here.
The Feature
To be able to use litellm with first party support for LM Studio!
Motivation, pitch
LM Studio is a very popular platform and they're the best way to get access to hugging face models locally. If possible having access to the LM Studio server through lite llm would be amazing
Twitter / LinkedIn details
No response