search

plasma-umass / cwhy

"See why!" Explains and suggests fixes for compile-time errors for C, C++, C#, Go, Java, LaTeX, PHP, Python, Ruby, Rust, and TypeScript
Apache License 2.0
280 stars 7 forks source link

Azure OpenAI and/or LiteLLM Proxy Support #65

Open dcieslak19973 opened 3 months ago

dcieslak19973 commented 3 months ago

Would be nice to allow users to specify either a LiteLLM Proxy instance or their own Azure OpenAI instance in addition to support for direct OpenAI and AWS Claude

emeryberger commented 3 months ago

This is possible; we are using LiteLLM in https://github.com/plasma-umass/ChatDBG/. If there's sufficient interest, we will add it (please thumbs-up this issue).

dcieslak19973 commented 2 months ago

Thanks; I think I see in the code that you may now be using litellm; less than clear what I would need to set (either environment variable-wise or cli-argument-wise) to connect to my LiteLLM proxy instance...

nicovank commented 2 months ago

I expect we also need to change the code a bit just to let that parameter pass through to the underlying litellm package. I’ll give it a look.

nicovank commented 2 months ago

So from what I understand, the proxy more or less gives the same functionality as having LiteLLM in the code namely abstracting out which service to request from. Do you have the possibility of directly querying the underlying service? e.g. can you specify AZURE_API_KEY, AZURE_API_BASE, and AZURE_API_VERSION directly in the environment? Or are you bound by the proxy = everything must go through the proxy? I assumed Azure OpenAI because that was in the title, but same question if it was another provider.

We currently do a hard check for either OpenAI or AWS credentials, and only pass those through to LiteLLM. We should be able to allow anything LiteLLM already allows instead.