.Net: Unable to auto invoke native functions

[mhill601]

Describe the bug Unable to auto invoke native functions using either ChatCompletionService or InvokePromptAsync. The kernel is aware of the plugin, it will reference it in the response (e.g. "If you want to do XXX call the PluginName::FunctionName").

To Reproduce Steps to reproduce the behavior:

I've tried a lot of different code, but here is my most recent attempt which is copied from semantic kernel documentation:

            ChatHistory history = new();
            history.AddUserMessage(prompt);

            // Enable auto function calling
            OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
            {
                ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
            };

            // Get the response from the AI
            var result = _chatService.GetStreamingChatMessageContentsAsync(
                                history,
                                executionSettings: openAIPromptExecutionSettings,
                                kernel: _kernel);

            // Stream the results
            string fullMessage = "";
            var first = true;
            await foreach (var content in result)
            {
                if (content.Role.HasValue && first)
                {
                    first = false;
                }
                fullMessage += content.Content;
            }

            ChatResponseDTO responseDTO = new ChatResponseDTO();
            responseDTO.Answer = fullMessage;
            return responseDTO; 

I've tried several different plugins as well, here is the most recent example:

using System;
using System.ComponentModel;
using Microsoft.SemanticKernel;

namespace ForvisAI.Plugins;

public sealed class WeatherPlugin
{
    [KernelFunction, Description("Get the weather in a city")]
    public static string WeatherCitySearch(
        [Description("The name of the city we want to know the weather of")] string cityName
    )
    {
        return "It's an unbearable 102 degrees.";
    }
}

Expected behavior When auto invoke is configured, the kernel functions should be auto invoked.

Platform Microsoft.SemanticKernel version 1.5.0

Azure model: gpt-35-turbo

[Krzysztof318]

Try again with latest changes, maybe https://github.com/microsoft/semantic-kernel/pull/5301 this fixes your issue also. If still not work show us full code with kernel initialization and user prompt.

[mhill601]

I've tried again with #5301 and am still having the same issue. In order to simplify this further I took the unit test from that commit, and modified it to work with my kernel. Below is all of the relevant code:

ChatService.cs

using Microsoft.SemanticKernel;
using ForvisAI.Models.DTOs;
using Microsoft.SemanticKernel.Connectors.OpenAI;

namespace ForvisAI.Services
{
    public class ChatService
    {
        private readonly Kernel _kernel;

        public ChatService(Kernel kernel)
        {
            _kernel = kernel;
        }

        /*
         * Copied unit test of _kernel.InvokePromptAsync
         */
        public async Task<ChatResponseDTO> Chat()
        {
            var promptFunction = KernelFunctionFactory.CreateFromPrompt(
                "Your role is always to return this text - 'A Game-Changer for the Transportation Industry'. Don't ask for more details or context.",
                functionName: "FindLatestNews",
                description: "Searches for the latest news.");

            _kernel.Plugins.Add(KernelPluginFactory.CreateFromFunctions(
                "NewsProvider",
                "Delivers up-to-date news content.",
                new[] { promptFunction }));

            OpenAIPromptExecutionSettings settings = new() { ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions };
            var result = await _kernel.InvokePromptAsync("Show me the latest news as they are.", new(settings));

            return new ChatResponseDTO()
            {
                Answer = result.ToString()
            };
        }
    }
}

Running this code returns an output of:

{
    "answer": "I'm sorry, but as an AI language model, I don't have the ability to display live news updates. However, I can suggest some news sources or websites that you can check for the latest news such as CNN, BBC, Reuters, The New York Times, and many more. If you have any specific topic or category in mind, let me know and I can try to provide more tailored recommendations for you.",
    "policyArticles": null
}

Kernel initialization is done in Program.cs

using ForvisAI;
using ForvisAI.Services;
using Microsoft.EntityFrameworkCore;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Memory;
using Microsoft.SemanticKernel.Connectors.Weaviate;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using ForvisAI.Settings;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.Identity.Web;
using ForvisAI.Plugins;
using pbc.Services;
using pbc.Models.Contexts;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAdB2C"));

builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

builder.Services.AddScoped<PolicyService>();
builder.Services.AddScoped<ChatService>();

//add options
builder.Services.AddOptions();
builder.Services.Configure<AppSettings>(builder.Configuration.GetSection("App"));

//Setup semantic kernel
var azureConfig = builder.Configuration.GetSection("App:AzureOpenAiConfig").Get<AzureOpenAiConfig>();

builder.Services.AddTransient<Kernel>(sp =>
{
    var semantic = Kernel.CreateBuilder();
    semantic.Services.AddLogging(c => c.AddConsole().SetMinimumLevel(LogLevel.Information));

    semantic.AddAzureOpenAIChatCompletion(
        azureConfig.Deployment,
        azureConfig.Endpoint,
        azureConfig.Key,
        modelId: azureConfig.ModelName
    );

    //Add the engage plugin
    semantic.Services.AddDbContext<PBCContext>(options => options.UseSqlServer(builder.Configuration.GetConnectionString("Project"), b => b.MigrationsAssembly("pbc.Server")));
    semantic.Services.AddDbContext<DataSpringContext>(options => options.UseSqlServer(builder.Configuration.GetConnectionString("DataSpring")));
    semantic.Services.AddPBCServices();
    semantic.Services.AddHttpContextAccessor();
    //semantic.Plugins.AddFromType<EngagePlugin>();
    semantic.Plugins.AddFromType<WeatherPlugin>();

    return semantic.Build();
});

//https://blog.brakmic.com/intro-to-semantic-kernel-part-four/
//https://learn.microsoft.com/en-us/semantic-kernel/get-started/quick-start-guide?tabs=Csharp
//https://github.com/microsoft/semantic-kernel/blob/main/dotnet/notebooks/06-memory-and-embeddings.ipynb
#pragma warning disable SKEXP0003, SKEXP0029, SKEXP0011
builder.Services.AddTransient<SemanticTextMemory>(sp => {
    string weaviateConn = builder.Configuration.GetSection("App:Weaviate:Url").Value ?? "";
    var embeddingModel = new AzureOpenAITextEmbeddingGenerationService(
        azureConfig.EmbeddingDeployment,
        azureConfig.Endpoint,
        azureConfig.Key,
        modelId: azureConfig.EmbeddingModelName
    );

    var vectorStore = new MemoryBuilder()
        .WithTextEmbeddingGeneration(embeddingModel)
        .WithWeaviateMemoryStore(weaviateConn, null, null);

    return (SemanticTextMemory)vectorStore.Build();
});

builder.Services.AddDbContext<ProjectContext>(options => options.UseSqlServer(builder.Configuration.GetConnectionString("Project")));

builder.Services.AddCors();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

//Temporary
app.UseCors(builder => builder
    .AllowAnyOrigin()
    .AllowAnyMethod()
    .AllowAnyHeader());

app.UseAuthentication();
app.UseAuthorization();

app.MapControllers();

app.Run();

Probably not relevant, but here is the controller:

using ForvisAI.Models.DTOs;
using ForvisAI.Services;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;

namespace ForvisAI.Controllers
{
    [Authorize]
    [ApiController]
    [Route("chat")]
    public class ChatController : ControllerBase
    {
        public readonly ChatService _chatService;

        public ChatController(ChatService chatService)
        {
            _chatService = chatService;
        }

        [HttpPost]
        public async Task<IActionResult> Chat(ChatRequestDTO req)
        {
            var res = await _chatService.Chat();
            return Ok(res);
        }
    }
}

And here is a screenshot when I debug it:

[Krzysztof318]

I think it can be too difficult to understand by gpt3.5 try with gpt-4-turbo-preview again. If it still doesn't help I will verify it tomorrow.

[markwallace-microsoft]

@mhill601 What region do you have the model deployed in?

[mhill601]

The model is deployed in East US. I switched to West US and it started working as expected!

[markwallace-microsoft]

@mhill601 thanks for confirming, I'm going to close this issue as it's not a Semantic Kernel problem

[bryantlin]

@mhill601, I think I am running to the similar problem you had. Just wanted to check that you said "The model is deployed in East US. I switched to West US...", did you mean the location of your Azure OpenAI instance as I was trying to find the location of model but can't find it?

Thanks you for your clarification in advance!