search

jared-yu-hcltech / custom-genai-chatbot

0 stars 0 forks source link

Add interoperability for the LLM model API #2

Closed jared-yu-hcltech closed 1 month ago

jared-yu-hcltech commented 2 months ago

Many clients rely on Azure Open AI, how can we create a version that utilizes Azure OpenAI in place of Gemini? It would be good to develop some sort of API so that users can switch between models, potentially using multiple models from different systems (e.g., Chat GPT, Gemini, Llama, etc.).

jared-yu-hcltech commented 2 months ago

Have been doing some testing with Azure's Open AI JS SDK https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart?tabs=command-line,python-new&pivots=programming-language-javascript. Having difficulty with switching out the model/API used with Gemini currently, but making some progress.

The code seems to be restricted largely to NewPrompt.jsx, but there are likely some other issues elsewhere. It would be good to build out a connector so that we can switch and use different models as desired. This would also work well with #12 and #15.

jared-yu-hcltech commented 2 months ago

As of 64161c4, we now can talk to Azure Open AI instead via the JS SDK. However, there is not really a good API yet to switch from one model to the other. Currently, the code for Gemini is commented out. It's worth noting too that the way that the two models are integrated into the system are slightly different.

With Gemini, we initialize a chat and then add another message. For Azure Open AI, we are initializing the entire history at once while also including the latest user message. This might make it a bit tricky, but we still need to develop an API for them all, as we may want to include things like Llama, Mistral, and possibly some fine tuned models.

jared-yu-hcltech commented 1 month ago

With 1bb9fbe we now have a menu that allows users to choose, but does not yet include real functionality.

jared-yu-hcltech commented 1 month ago

12 is fairly smooth now. We have a basic API that works for our current setup. It's slightly hard-coded, but it's more akin to setting up a config. When we add more advanced integrations like RAG, we can open a new issue and further reformat/update the existing API.