search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.25k stars 1.93k forks source link

[FEATURE REQ] Support overriding the endpoint for NonAzureOpenAIClientService #39742

Closed viswanathk closed 1 month ago

viswanathk commented 2 months ago

Is your feature request related to a problem? Please describe. The current OpenAI API is used by multiple providers, and Azure SDK is the only "official" Java SDK that supports this API. However, the endpoint is hardcoded in this file limiting it's usefulness to be used with other providers.

Describe the solution you'd like Allow the client to override the endpoint along with other attributes when constructing the NonAzureOpenAIClientImpl object so it can be used internally within companies (if there's an internal gateway that proxies that request), or use this with other providers.

Describe alternatives you've considered The other option would be for another implementation of the API in java, which is completely needless.

Additional context NA

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

viswanathk commented 2 months ago

Also, I am happy to contribute this change if the team is okay with the described solution.

alzimmermsft commented 2 months ago

@mssfang could you follow-up

mssfang commented 2 months ago

@viswanathk Recently we get this feature request and waiting to see if we should support it in the SDK. Will keep you updated.

https://github.com/Azure/azure-sdk-for-java/pull/39659

viswanathk commented 2 months ago

@mssfang any timeline for this? :)

mssfang commented 2 months ago

Unfortunately, we will not support non-OpenAI models in this SDK, azure-ai-openai as per our partnership with OpenAI. The most major models are available as first-class Azure Host Endpoints, such as Llama2, Cohere and Mistra, via Azure MaaS (Azure AI Studio Model Catalog). They can also host most Hugging Face models or your own models in the future via Azure MaaP(AML) in the future. We will provide an ("OpenAI like") Model Inference API spec for completions to MaaS in the near future.

Worth to mention here that this SDK supports proxy feature in case someone not aware of it.