Can you add the Azure OpenAI interface?

[AnujMuth]

Hello, can you add the Azure OpenAI interface? In this way, you can use the API KEY of Azure OpenAI. The KEY of Azure OpenAI includes three functions: language APIS, Dall-e APIS, and Whisper APIS. Can you also add the model option of GPT4-32K? Thank you.

[DevEmperor]

I have just added GPT-4-32K. Thank you for the suggestion. As for the Azure interface to OpenAI, that will probably take a little longer, as Microsoft works with a completely different library that I first have to integrate into the project and test.

[AnujMuth]

Thank you, I'm looking forward to the inclusion of Azure OpenAI version. I have another question: I noticed that every time the AI provides an answer, it outputs all the text on the page. Is it possible to make it display with typewriter-like typing effect? Would this reduce the response time of the AI servers and TTS?

[DevEmperor]

The typewriter effect is possible, but not faster than the normal API. The segments are then simply delivered individually and not together. I have had the idea of adding this feature for some time and will put it on the to-do list. (However, the TTS can only be started with the entire text...)

[ishaan-jaff]

Hi @DevEmperor @AnujMuth - I believe we can make this easier I’m the maintainer of LiteLLM - we allow you to deploy a LLM proxy to call 100+ LLMs in 1 format - Azure OpenAI, PaLM, Bedrock, OpenAI, Anthropic etc https://github.com/BerriAI/litellm/tree/main/openai-proxy.

If this looks useful (we're used in production)- please let me know how we can help.

Usage

Azure request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "azure/<your-deployment-name>",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

gpt-3.5-turbo request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

claude-2 request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "claude-2",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

[DevEmperor]

@ishaan-jaff Thank you for drawing attention to this project. This looks like exactly the solution we actually need. Unfortunately, I don't know much about deploying a proxy on servers. Is there a publicly available instance of your project?

[ishaan-jaff]

@DevEmperor we have 1 click deploys for the proxy - you can add your keys to the deployed instance https://docs.litellm.ai/docs/simple_proxy#deploy-on-google-cloud-run

You just need to click on the button and it's deployed on google cloud run for you

[AnujMuth]

@DevEmperor we have 1 click deploys for the proxy - you can add your keys to the deployed instance https://docs.litellm.ai/docs/simple_proxy#deploy-on-google-cloud-run

You just need to click on the button and it's deployed on google cloud run for you

Thank you, can this be deployed to AZURE above? How to do it specifically?

[DevEmperor]

@ishaan-jaff @AnujMuth Thank you for pointing this out. This looks very promising, I will look into it in the next few days and try to rework the code so that it works with Azure OpenAI (and possibly other models like PaLM).

[DevEmperor]

@AnujMuth Hey, I'm getting back to you. I've been reading through the Azure OpenAI documentation and unfortunately access is only granted on request. I'm not an organisation, so unfortunately I don't get access and so I can't implement and test the interface. I am very sorry about that. Are there any advantages at all compared to the conventional API of OpenAI?

[AnujMuth]

Azure openai has advantages over openai in many places. I have permission to use azure openai and can give you a test KEY. If you need it, I can send it to your email separately for testing.

[DevEmperor]

That would be great. Could you send me a test key to contact@devemperor.net ? Then I will will try to implement Azure OpenAI. :)

[DevEmperor]

Hey, I'll get back to you about your feature requests :)

About the typewriter effect: The API of OpenAI offers the function of streams, but only paragraphs are transmitted individually, which does not create a real typewriter effect. In addition, there is no more information about the token consumption, which would limit other features in my app. As the responses don't arrive any faster anyway and the feature is therefore rather redundant, I decided not to implement it at this cost.

The same applies to the connection to Azure OpenAI: the feature would hardly be used by any users, but I would have to integrate a completely new library, which would tend to flood the entire project. With WristAssist, I want to provide an app for everyday use and normal users.

I'm sorry that I won't be integrating both features in the near future.