search

enricoros / big-AGI

Generative AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. It features AI personas, AGI functions, multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.
https://big-agi.com
MIT License
5.27k stars 1.2k forks source link

[Roadmap] AzureAI support for large language models #608

Open wxwatcher2004 opened 1 month ago

wxwatcher2004 commented 1 month ago

Why Llama 3.1 405b just released and one of the only way to get access to it is through AzureAI hosting it.

Description Support AzureAI for non OpenAI models as a cloud service.

Requirements Here is the current token error when I try to use AzureAI on my Llama 3.1 server. [Issue] OpenAI/azure: Bad Request - "status": "Auth token must be passed as a header called Authorization"

enricoros commented 1 month ago

@wxwatcher2004 to replicate this, how do I setup a service, and where do you see the error? Please add some details so I can reproduce easily.

wxwatcher2004 commented 1 month ago

So the issue occurs in the models page and I think is an issue with how Azure works. They have a connection side to OpenAI, I followed the instructions you have and found that to work correctly. The other open source models use serverless options. Best I can tell these are accessed through the model deployment URL and Key. I also tried https://x.cognitiveservices.azure.com/ configuration but that only showed the GPT4 model as well.

working OpenAI connection image Not working Llama 3.1 connection image Azure model setting screen image