.Net: Bug: ChatCompletionAgent.InvokeStreamingAsync(...) method when called with data source fails when Instructions not empty

[PanClifton]

Describe the bug When InvokeStreamingAsync(...) is called on ChatCompletionAgent who has AzureSearchChatDataSource connected to it fails with 400 BadRequest

To Reproduce Steps to reproduce the behavior:

Setup csproj

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net8.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Azure.Identity" Version="1.13.1" />
        <PackageReference Include="Azure.Search.Documents" Version="11.6.0" />
        <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.1" />
        <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" Version="8.0.1" />
        <PackageReference Include="Microsoft.SemanticKernel.Agents.Core" Version="1.29.0-alpha" />
        <PackageReference Include="Microsoft.SemanticKernel" Version="1.29.0" />
        <PackageReference Include="Microsoft.SemanticKernel.Abstractions" Version="1.29.0" />
        <PackageReference Include="Microsoft.SemanticKernel.Connectors.OpenAI" Version="1.29.0" />
    </ItemGroup>

    <ItemGroup>
        <ProjectReference Include="..\HeiDi.Sandbox.Utils\HeiDi.Sandbox.Utils.csproj" />
    </ItemGroup>

</Project>

failing code snippet

#pragma warning disable SKEXP0001
#pragma warning disable SKEXP0010
#pragma warning disable SKEXP0110
#pragma warning disable SKEXP0020
#pragma warning disable AOAI001

using Azure.AI.OpenAI.Chat;
using Azure.Search.Documents;
using HeiDi.Sandbox.Utils.Providers;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Agents;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.AzureOpenAI;
using System.Text.Json;

var configurationProvider = ConfigurationsProvider.CreateNew();

var aiSearchConfigration = configurationProvider.AiSearchConfiguration;
var openAiConfigration = configurationProvider.OpenAiConfigration;

var datasource = new AzureSearchChatDataSource
{
    Endpoint = new Uri(aiSearchConfigration.Endpoint),    
    Authentication = DataSourceAuthentication.FromApiKey(aiSearchConfigration.ApiKey),
    IndexName = "contracts-index",
    Filter = SearchFilter.Create($"source_id eq 'generalknowledge'"),
};

var kernel = Kernel.CreateBuilder()
      .AddAzureOpenAIChatCompletion(
          openAiConfigration.ModelDelpoymentName,
         openAiConfigration.Endpoint,
         openAiConfigration.ApiKey)
      .Build();

var chatHistory = new ChatHistory();

var question = "Could you describe the change request process including all steps?";

Console.WriteLine($"User > {question}");

chatHistory.AddUserMessage(question);

var agent = new ChatCompletionAgent()
{
    Description = "You are a frendly assisant for user",
    // fail when Instructions has value
    Instructions = "Use in your answers the same language than the user has used in the question",
    // it will not fail with
    /*

    Instructions = "",

     */
    Name = "open_ai",
    Kernel = kernel,
    Arguments = new KernelArguments(new AzureOpenAIPromptExecutionSettings()
    {
        TopP = 0.5,
        Temperature = 0.5,
        MaxTokens = 1000,
        AzureChatDataSource = datasource,
    }),
};

var citations = new List<ChatCitation>();

await foreach (var content in agent.InvokeStreamingAsync(chatHistory))
{
    var contentReult = content.Content ?? string.Empty;
    if (!string.IsNullOrEmpty(contentReult))
        Console.WriteLine($"Asistatnt > {contentReult}");

    var update = content.InnerContent as OpenAI.Chat.StreamingChatCompletionUpdate;
    var messageContext = update?.GetMessageContext();

    if (messageContext?.Citations is not null)
    {
        citations.AddRange(messageContext?.Citations);
    }
}

var options = new JsonSerializerOptions { WriteIndented = true };
var json = JsonSerializer.Serialize(citations.Select(x => new Citation(x.Title, x.FilePath, x.Content)).Distinct(), options);

Console.WriteLine($"Tool > Citations: {Environment.NewLine}{json}");
Console.WriteLine("end of the line");
Console.ReadKey();

public record Citation(string Titile, string FileSource, string Content);

Expected behavior InvokeStreamingAsync(...) method should start stream

Screenshots If applicable, add screenshots to help explain your problem.

Platform

• OS: windows
• IDE: Visual Studio
• Language: C#
• Source: sources are given in setup in to reproduce step

Additional context Add any other context about the problem here.

[dmytrostruk]

@PanClifton Thanks for creating this issue. Unfortunately, HTTP error message is not included in the exception, so I had to add HTTP logging handler to see what an actual response was. Here is an example of such logging handler, so you can use it for your own purposes in the future: https://github.com/microsoft/semantic-kernel/blob/f5facce9ec0dc86e97a13dad0ac4c2476f218bb7/dotnet/src/InternalUtilities/samples/InternalUtilities/BaseTest.cs#L150

Based on logging, there is a following error message in response: Validation error at #/messages/0/name: Extra inputs are not permitted. I removed Name = "open_ai", from ChatCompletionAgent in your code example and it worked as expected. There could be a case that Azure OpenAI Chat Completion with data service doesn't support additional parameters in messages such as agent name.

Please let us know if that works for you. Thanks!

[PanClifton]

@dmytrostruk thanks for the fast response

It worked, thanks