.Net: Bug: Semantic Kernel - DotNet - Issues With AzureChatExtensionsOptions implementation inside OpenAIPromptExecutionSettings

[raeesgillani]

Describe the bug When defining executionSettings (specifically with AzureChatExtensionsOptions inside OpenAIPromptExecutionSettings) for creating a function from prompt, then passing it into CreatePlanAsync and in turn into InvokeAsync. I am finding no search results are being returned and passed into the prompt that gets sent to the model. And the responses received are mostly 'I don't know'. This is not always the case, sometimes it works, but once it gets stuck in a cycle of 'I don't know' it just fails. Or at some points it even generates the following errors:

Unhandled exception. Microsoft.SemanticKernel.KernelException: [HallucinatedHelpers] The plan references hallucinated helpers: Helper 'join' ---> HandlebarsDotNet.HandlebarsRuntimeException: Template references a helper that cannot be resolved. Helper 'join' at HandlebarsDotNet.Helpers.MissingHelperDescriptor.Invoke(HelperOptions& options, Context& context, Arguments& arguments) at HandlebarsDotNet.Helpers.MissingHelperDescriptor.HandlebarsDotNet.Helpers.IHelperDescriptor<HandlebarsDotNet.HelperOptions>.Invoke(HelperOptions& options, Context& context, Arguments& arguments) at HandlebarsDotNet.Helpers.LateBindHelperDescriptor.Invoke(HelperOptions& options, Context& context, Arguments& arguments) at HandlebarsDotNet.Helpers.LateBindHelperDescriptor.HandlebarsDotNet.Helpers.IHelperDescriptor<HandlebarsDotNet.HelperOptions>.Invoke(HelperOptions& options, Context& context, Arguments& arguments) at lambda_method50(Closure, EncodedTextWriter&, BindingContext) at HandlebarsDotNet.HandlebarsEnvironment.<>c__DisplayClass19_0.<Compile>b__0(TextWriter writer, Object context, Object data) at HandlebarsDotNet.HandlebarsEnvironment.<>c__DisplayClass20_0.<Compile>b__0(Object context, Object data) at Microsoft.SemanticKernel.PromptTemplates.Handlebars.HandlebarsPromptTemplate.RenderAsync(Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken) at Microsoft.SemanticKernel.Planning.Handlebars.HandlebarsPlan.InvokeCoreAsync(Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)

To Reproduce Steps to reproduce the behavior:

1. Using Microsoft.SemanticKernel, Microsoft.SemanticKernel.Planning, and using Microsoft.SemanticKernel.Connectors.OpenAI
2. Create a console app as follows:

` // Create Kernel var kernel = Kernel.CreateBuilder() .AddAzureOpenAIChatCompletion( deploymentName: config["YourSettings:OpenAIDeploymentId"]!, endpoint: config["YourSettings:OpenAIEndpoint"]!, apiKey: config["YourSettings:OpenAIKey"]!) .Build();

        // Azure Chat Extension Configuration
        AzureSearchChatExtensionConfiguration azureSearchExtensionConfiguration = new()
        {
            SearchEndpoint = new Uri(config["YourSettings:AzureAISearchEndpoint"]!),
            Authentication = new OnYourDataApiKeyAuthenticationOptions(config["YourSettings:AzureAISearchKey"]!),
            IndexName = config["YourSettings:AzureAIIndex"]!,
            QueryType = AzureSearchQueryType.VectorSemanticHybrid,
            SemanticConfiguration = "default",
            VectorizationSource = new OnYourDataDeploymentNameVectorizationSource(config["YourSettings:TextEmbeddingModelDeploymentName"]!),
            DocumentCount = 20,
            ShouldRestrictResultScope = false,
            Strictness = 4,
            FieldMappingOptions = new AzureSearchIndexFieldMappingOptions()
            {
                FilepathFieldName = "FilePath",
                TitleFieldName = "ChunkTitle",
                UrlFieldName = "Url",
            }
        };

        var contentFieldNames = new List<string>
        {
            "Bid", "Client", "BidType", "ScopeOfWork", "FileName", "Content"
        };
        foreach (var fieldName in contentFieldNames)
        {
            azureSearchExtensionConfiguration.FieldMappingOptions.ContentFieldNames.Add(fieldName);
        }
        azureSearchExtensionConfiguration.FieldMappingOptions.VectorFieldNames.Add("contentVector");

        var chatExtensionsOptions = new AzureChatExtensionsOptions { Extensions = { azureSearchExtensionConfiguration } };

        // Prompt for the Semantic Function
        var skPrompt = @$"
        You are a bid writer. Your responses should be very highly detailed. You must use British English.
        You can say 'Error: I am unable to find any documents.' if no relevant documents are found in the index.
        Be truthful, don't lie.
        Response should not be harmful.
        Do not include any references.";

        #pragma warning disable SKEXP0010 // Suppress evaluation-only diagnostic
        var executionSettings = new OpenAIPromptExecutionSettings 
        {
            MaxTokens = 4000,
            Temperature = 0,
            TopP = 0.5,
            AzureChatExtensionsOptions = chatExtensionsOptions
        };
        #pragma warning restore SKEXP0010 // Restore evaluation-only diagnostic

        kernel.CreateFunctionFromPrompt(skPrompt, executionSettings, "BidResponse");

        // HandlebarsPlanner Setup
        #pragma warning disable SKEXP0060 // Suppress evaluation-only diagnostic
        var planner = new HandlebarsPlanner();
        #pragma warning restore SKEXP0060 // Restore evaluation-only diagnostic

        // Define the ASK
        var ask = "How does X handle Social Value? Provide ten specific examples and include the names of any specific clients involved and dates.";

        // Create a plan
        #pragma warning disable SKEXP0060 // Suppress evaluation-only diagnostic
        var plan = await planner.CreatePlanAsync(kernel, ask);
        #pragma warning restore SKEXP0060 // Restore evaluation-only diagnostic

        // Execute the plan
        #pragma warning disable SKEXP0060 // Suppress evaluation-only diagnostic
        var result = await plan.InvokeAsync(kernel, new KernelArguments());
        #pragma warning restore SKEXP0060 // Restore evaluation-only diagnostic

        // Output the result
        Console.WriteLine("Plan results:\n");
        Console.WriteLine(result);
    }
}

}`

3. dotnet run returns non-sensical examples that are essentially placeholders for the text from Azure AI Search. This is a response when asking for 10 examples, but if asking for say 5 examples, it sometimes returns a decent response:

   
   Plan results:

Example 1 of Social Value handled by X.\nClient: Client 1, Date: 2023-10-1 Example 2 of Social Value handled by X.\nClient: Client 2, Date: 2023-10-2 Example 3 of Social Value handled by X.\nClient: Client 3, Date: 2023-10-3 Example 4 of Social Value handled by X.\nClient: Client 4, Date: 2023-10-4 Example 5 of Social Value handled by X.\nClient: Client 5, Date: 2023-10-5 Example 6 of Social Value handled by X.\nClient: Client 6, Date: 2023-10-6 Example 7 of Social Value handled by X.\nClient: Client 7, Date: 2023-10-7 Example 8 of Social Value handled by X.\nClient: Client 8, Date: 2023-10-8 Example 9 of Social Value handled by X.\nClient: Client 9, Date: 2023-10-9 Example 10 of Social Value handled by X.\nClient: Client 10, Date: 2023-10-10

**Expected behavior**
Response should have the values from the search being passed through. When searching inside AI Search via Azure Portal for same text, it does return results fine, just not inside Semantic Kernel.

**Platform**
 - OS: Windows
 - IDE: VS Code
 - Language: C#
 - Source:
    <PackageReference Include="Microsoft.SemanticKernel" Version="1.15.0" />
    <PackageReference Include="Microsoft.SemanticKernel.Planners.Handlebars" Version="1.15.0-preview" />

**Additional context**
Model is GPT4o GA, AI Search indexes created using Ada-Text-Embedder-Large.
Additionally, please advise on the following:
Is it because all the AI Search parts inside Semantic Kernel are in preview? Same with the Planner? - if so, when will it be out of preview?
Are there any limitations that I should be aware of with AI Search inside Semantic Kernel?

[dmytrostruk]

@raeesgillani Thanks for reporting this issue! Based on provided code and prompt, it looks like you don't need to use planner. At least, I don't see any other plugins imported, so AI can generate a plan and execute these plugins. If you want to implement chat experience using your data in Azure AI Search, instead of planner you can invoke the same prompt by using kernel.InvokePromptAsync(prompt, new KernelArguments(executionSettings)) or kernel.InvokeAsync(function, arguments), here is an example: https://github.com/microsoft/semantic-kernel/blob/8140684f241022335b67d48490857a50dc48c041/dotnet/samples/Concepts/ChatCompletion/AzureOpenAIWithData_ChatCompletion.cs#L99-L105

Is it because all the AI Search parts inside Semantic Kernel are in preview? Same with the Planner?

No, even if some features are in preview/marked as experimental, they still should work. It is identifier that we may introduce breaking changes to this functionality if it needs to be improved.

Are there any limitations that I should be aware of with AI Search inside Semantic Kernel?

Based on your case, it should work as expected when using kernel directly. One limitation is related to using AI Search and Plugins together at the same time, currently it's not supported: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/openai/Azure.AI.OpenAI#use-your-own-data-with-azure-openai

NOTE: The concurrent use of Chat Functions and Azure Chat Extensions on a single request is not yet supported. Supplying both will result in the Chat Functions information being ignored and the operation behaving as if only the Azure Chat Extensions were provided. To address this limitation, consider separating the evaluation of Chat Functions and Azure Chat Extensions across multiple requests in your solution design.

Thank you!

[raeesgillani]

Thank you @dmytrostruk that makes sense.