[FEATURE] Enhance Azure OpenAI Integration - Azure Cognitive Search Building Block or similar

[sirsimonson]

Describe the feature you'd like When working with Azure OpenAI Chat LLM or Chat Model Step I'm missing optional params that are sent to the endpoint. For example when working with Azure OpenAI and Cognitive Search as state here: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data

There are 2 main topics where the current Building Block is limited.

• Custom URL -> A ChatGPT deployment with Cognitive Search enabled has a different URL (https://{instance-name}.openai.azure.com/openai/deployments/{deployment-name}/extensions/chat/completions?api-version=2023-08-01-preview) /extensions/ is an additional path param in this URL
• extra arguments -> The Azure ChatGPT completions endpoint with Cognitive Search enabled has additional Request Body Params. In this case I'm not able to add a "dataSource" prop to the request in order to build a request body as below

  parameters = {
  "max_tokens": 500,
  "temperature": 0.5,
  "top_p": 0.3,
  "stream": False,
  "dataSources": [
      {
          "type": "AzureCognitiveSearch",
          "parameters": {
              "endpoint": search_endpoint,
              "key": search_admin_key,
              "indexName": index_name,
              "topNDocuments": 1,
              "queryType": "simple",
              #"intent": product,
              "semanticConfiguration": "default",
              "roleInformation": prompt,
              "inScope": True,
              "filter": f"search.ismatch('\"{product}\"','{AZURE_SEARCH_TITLE_COLUMN}')",
              "fieldsMapping": {
                  "contentField": AZURE_SEARCH_CONTENT_COLUMNS.split("|") if AZURE_SEARCH_CONTENT_COLUMNS else [],
                  "titleField": AZURE_SEARCH_TITLE_COLUMN if AZURE_SEARCH_TITLE_COLUMN else None,
                  "urlField": AZURE_SEARCH_URL_COLUMN if AZURE_SEARCH_URL_COLUMN else None,
                  "filepathField": AZURE_SEARCH_FILENAME_COLUMN if AZURE_SEARCH_FILENAME_COLUMN else None,
                  "vectorFields": AZURE_SEARCH_VECTOR_COLUMNS.split("|") if AZURE_SEARCH_VECTOR_COLUMNS else []}
          }
      }
  ],
  "messages": [
      {"role": "user", "content": f"{product} {string}"}
  ]
  }

I think it's enough to add the ability to override the URL as stated here: https://github.com/FlowiseAI/Flowise/issues/960 as well as adding BaseOptions to the Azure Node in order to pass thourgh "additional_kwargs" At least thats all I could find out when doing a little search on this in the source code of langchain and flowise

Looking forward to hear from you!

[HenryHengZJ]

Thought it would be an easy change, but seems more to it. Reason being we are using @openai npm package for Azure, and this does not have the extension chat capability, we need to switch to use @azure/openai for that

[sirsimonson]

Feel free to point out more guidance here, I'd love to support but I'm still new in the field (AI & LangChain)..

What do you mean by not having the capability? As you can see here in the test the baseURL could be overwritten inside the OpenAIEndpointConfig https://github.com/langchain-ai/langchainjs/blob/4e9e23137de8d2bc61dfbcf05e8e35a8f1e89ef5/langchain/src/util/tests/azure.test.ts#L49

I haven't yet fully understood how the client is constructed but I wonder if we might find a way to hook in here.

[HenryHengZJ]

if you have an azure api key specified, the baseUrl will not be used. Instead this will be used ${azureOpenAIBasePath}/${azureOpenAIApiDeploymentName} - https://github.com/langchain-ai/langchainjs/blob/4e9e23137de8d2bc61dfbcf05e8e35a8f1e89ef5/langchain/src/util/azure.ts#L45

[sirsimonson]

Ok thanks for pointing that out. Looks like I'm on the right path then. I tried locally and directly edited node_modules and came across the same line. I might dive a bit deeper when I find the time.