.Net Bug: Bing Connector and Azure AI Search Issue

[Cobra86]

Describe the bug When using the Web Search Engine plugin and Bing Connector in combination with Azure AI Search in the same request, I occasionally encounter a 400 Bad Request error, although it works at other times.

Error Message: "Invalid chat message detected: message content must be string"

Platform

• AI Model: Azure OpenAI GPT4o
• OS: Windows
• IDE: Visual Studio
• Language: C#
• Source: NuGet package version 1.17.1

[evchaki]

@Cobra86 can you provide more details on this?

[Cobra86]

Sure, I'm trying to use the Bing connector and the Azure AI Search index in the same kernel build. Apologise, I can't share the actual code, so I've created this sample code instead. I'm also passing the same chat history.

The code works fine when I'm only calling bing search. However, when I start asking the AI about data in the index, it returns a data or 400 Bad Request error. I checked the meta data but i couldn't find something.

I remembered there was some limitation with Kernel plugins and Azure AI Search but not sure why sometime it works.

        // Configure Semantic Kernel

        ChatHistory chatHistory = new ChatHistory();
        var builder = Kernel.CreateBuilder();

        builder.AddAzureOpenAIChatCompletion(openAiSettings.ChatDeploymentName, endpoint: openAiSettings.Endpoint,
    apiKey: openAiSettings.ApiKey, serviceId: openAiSettings.ChatModelId);

        builder.Services.AddSingleton<IWebSearchEngineConnector>(new BingConnector(pluginSettings.BingApiKey));

        builder.Plugins.AddFromType<WebSearchEnginePlugin>();

        Kernel kernel = builder.Build();

        var prompt = "What's Semntic Kerenal ?";

        WriteLine($"\nQUESTION: \n\n{prompt}");

#pragma warning disable SKEXP0010
        var azureSearchExtensionConfiguration = new Azure.AI.OpenAI.AzureSearchChatExtensionConfiguration
        {
            SearchEndpoint = new Uri(openAiSettings.AISearchEndpoint),
            Authentication = new OnYourDataApiKeyAuthenticationOptions(openAiSettings.AISearchApiKey),
            IndexName = "contoso-products-with-weather-index"
        };
        var chatExtensionsOptions = new AzureChatExtensionsOptions { Extensions = { azureSearchExtensionConfiguration } };

        OpenAIPromptExecutionSettings settings = new()
        {
            ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
            Temperature = 0.7f,
            MaxTokens = 250,
            ChatSystemPrompt = "Return the response as json object and put url,summary",
            AzureChatExtensionsOptions = chatExtensionsOptions
        };

        chatHistory.AddUserMessage(prompt);

        var chatMessage = await kernel.Services.GetService<IChatCompletionService>().GetChatMessageContentAsync(chatHistory, settings);

        WriteLine($"\nANSWER: \n\n{chatMessage}");

[evchaki]

@markwallace-microsoft - FYI