.Net: Bug: Exception when using Plugin and AzureSearchChatExtensionConfiguration

[AlexSchlecht]

Describe the bug If you use a custom plugin while simultaneously using AzureSearchChatExtensions, an exception is thrown after the plugin method is executed. The plugin method itself is executed correctly. Without plugin but with AzureSearchChatExtension, it works all fine. Without AzureSearchChatExtension but with Plugin, it works all fine. (Of Course without my Data in Azure Blob-Storage)

To Reproduce A prompt which will trigger the Plugin-Method will cause an Exception. Here ist some sample-Code:

My Program.cs

using AzureSearch_Error;
await PluginAzureSearch_Error.Start();

My Code

using Azure.AI.OpenAI;
using Microsoft.Extensions.Configuration;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using SUMS3.Win.SUMS3System;

namespace AzureSearch_Error
{
    public class PluginAzureSearch_Error
    {
        public static IConfiguration configuration;
        public static void InitConfig()
        {
            var configBuilder = new ConfigurationBuilder()
                .SetBasePath(AppContext.BaseDirectory)
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                .AddUserSecrets<Program>(); 

            configuration = configBuilder.Build();
        }

        public static async Task Start()
        {
            InitConfig();

            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(
                configuration["deploymentName"]!,
                configuration["endpoint"]!,
                configuration["apiKey"]!
            );

            builder.Plugins.AddFromType<MyTestPlugin>("MyTestPlugin");

            Kernel kernel = builder.Build();

            var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();
            OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
            {
                ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,  //for Plugin

#pragma warning disable SKEXP0010

                //Ursache für einen Fehler mit Plugins!
                AzureChatExtensionsOptions = GetAzureChatExtensionsOptions()

#pragma warning restore SKEXP0010
            };

            var history = new ChatHistory();
            var frage = "send me an email";   //This prompt triggers the Plugin

            Console.Write("Du: ");
            Console.WriteLine(frage);

            history.AddUserMessage(frage);

            // Get the response from the AI
            //Here a error occurs, after (!) successful execution of Plugin-Method   <------------------ Error
            //This error only occurs with a prompt that triggers the plugin.

            var result = await chatCompletionService.GetChatMessageContentAsync(
                history,
                executionSettings: openAIPromptExecutionSettings,
                kernel: kernel);

            Console.WriteLine("KI: " + result);
            history.AddAssistantMessage(result.ToString());

            Console.WriteLine();
        }

        public static AzureChatExtensionsOptions GetAzureChatExtensionsOptions()
        {
            var azureSearchExtensionConfiguration = new AzureSearchChatExtensionConfiguration
            {
                SearchEndpoint = new Uri(configuration["searchEndpoint"]!),
                Authentication = new OnYourDataApiKeyAuthenticationOptions(configuration["authentication"]!),
                IndexName = configuration["indexName"]!
            };

            return new AzureChatExtensionsOptions
            {
                Extensions = { azureSearchExtensionConfiguration }

            };
        }
    }
}

Here my Plugin-Code (simple sample)

using Microsoft.SemanticKernel;
using System.ComponentModel;

namespace SUMS3.Win.SUMS3System
{
    public class MyTestPlugin
    {
        [KernelFunction("sendmail_to_me")]
        [Description("Sendet an den User eine email.")]
        [return: Description("Text über das Ergebnis des Mailversandes")]
        public string SendEmailToMe()
        {
            try
            {

                return "erledigt, die Mail wurde gesendet.";
            }
            catch (Exception ex)
            {
                return $"Fehler, Mail konnte nicht gesendet werden {ex.Message}";
            }
        }
    }
}

Screenshots

Platform

• OS: Windows
• IDE: Microsoft Visual Studio Enterprise 2022 (64-Bit) - Current, Version 17.9.6
• Language: C#
• Model: gpt-4o

[AlexSchlecht]

I forgot the detailed error message:

    Microsoft.SemanticKernel.HttpOperationException
      HResult=0x80131500
      Nachricht = Service request failed.
    Status: 400 (Bad Request)

    Content:
    {"error": {"requestid": "d784ca74-fca8-49e1-bdcc-1bdf80aeb25a", "code": 400, "message": "Invalid chat message detected: message content must be string"}}

    Headers:
    skip-error-remapping: REDACTED
    x-envoy-upstream-service-time: REDACTED
    apim-request-id: REDACTED
    x-ms-client-request-id: 5f6198ba-5064-48c3-9da1-f04ef4d1a229
    Strict-Transport-Security: REDACTED
    X-Content-Type-Options: REDACTED
    x-ms-region: REDACTED
    x-ratelimit-remaining-requests: REDACTED
    x-ratelimit-remaining-tokens: REDACTED
    Date: Tue, 03 Sep 2024 11:35:34 GMT
    Content-Length: 153
    Content-Type: application/json; charset=UTF-8

      Quelle = Microsoft.SemanticKernel.Connectors.OpenAI
      Stapelüberwachung:
       bei Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.<RunRequestAsync>d__55`1.MoveNext()
       bei Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.<GetChatMessageContentsAsync>d__35.MoveNext()
       bei Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.<GetChatMessageContentAsync>d__2.MoveNext()
       bei AzurOpenAI.LightPluginBeispiel.<Start>d__2.MoveNext() in C:\Users\a2002\source\GitHub\spielwieseVersuche\LightPluginBeispiel.cs: Zeile75
       bei Program.<<Main>$>d__0.MoveNext() in C:\Users\a2002\source\GitHub\spielwieseVersuche\Program.cs: Zeile5

      Diese Ausnahme wurde ursprünglich von dieser Aufrufliste ausgelöst:
        [Externer Code]

    Innere Ausnahme 1:
    RequestFailedException: Service request failed.
    Status: 400 (Bad Request)

    Content:
    {"error": {"requestid": "d784ca74-fca8-49e1-bdcc-1bdf80aeb25a", "code": 400, "message": "Invalid chat message detected: message content must be string"}}

    Headers:
    skip-error-remapping: REDACTED
    x-envoy-upstream-service-time: REDACTED
    apim-request-id: REDACTED
    x-ms-client-request-id: 5f6198ba-5064-48c3-9da1-f04ef4d1a229
    Strict-Transport-Security: REDACTED
    X-Content-Type-Options: REDACTED
    x-ms-region: REDACTED
    x-ratelimit-remaining-requests: REDACTED
    x-ratelimit-remaining-tokens: REDACTED
    Date: Tue, 03 Sep 2024 11:35:34 GMT
    Content-Length: 153
    Content-Type: application/json; charset=UTF-8

[evchaki]

@dmytrostruk - can you make sure the error message are clear on this.