.Net: Allow responses from Native Functions to be returned directly when using Automatic Function Calling

[sandeepvootoori]

Currently when using Auto Function Calling, SK always assumes response from a Native Function is a tool message. We do have some scenarios where its desired to be sent the response from a Native Function as an Assistant message. For example we have a Native Function which answers question in a very specific format because of legal reasons but when we use Automatic Function calling it passes Native Function response as a tool message and LLM changes the text in the subsequent LLM call.

[sandeepvootoori]

@matthewbolanos

[stephentoub]

I'm not quite following.

You're saying you're including a function in the kernel, asking for auto-function invocation, the LLM asks for that function to be invoked, it's invoked, but you don't want the result of that function sent back to the LLM? What do you want to happen?

[sandeepvootoori]

I want the result of the native function to be returned and Auto Function calling should end at that point because the goal has been reached for generating a response to the user. I have tried prompting it to end but it never stops execution at Native Function

[stephentoub]

I see, thanks. What I've been thinking should be done, and would likely be necessary for https://github.com/microsoft/semantic-kernel/issues/4300 as well, is some kind of callback that can be provided that would allow the caller to have much more fine-grained control over the auto-invocation, e.g. it would be provided with information like the KernelFunction, the KernelArguments, the number of functions that have already been invoked, etc., and it would get to determine whether to invoke the function or end the processing, and potentially even substitute a different function to invoke, tweak the arguments, etc. It needs some design work, though.

[bjsmiley]

@sandeepvootoori I'm running into the same problem. My use case is to have the planner respond with a question when it doesnt have enough information from the user to invoke the function it actually wants to use. A hacky workaround I have is to use a CancellationTokenSource & Kernel.Data to exit early & reply with the question. I am very open to alternatives.

        var cts = new CancellationTokenSource();
        kernel.Data["stop"] = cts;
        kernel.Data["question"] = new Reference<string>(null); // Reference is just a wrapper object to persist the inner value across Kernel.Clone() calls

        FunctionCallingStepwisePlannerResult? result = null;
        try
        {
            result = await planner.ExecuteAsync(kernel, ask, cts.Token);
        }
        catch (OperationCanceledException) when (cts.IsCancellationRequested)
        {
            if(kernel.Data.TryGetValue("question", out var q) && q is Reference<string> { Inner: not null } question)
            {
                return Question.Inner;
            }
        }
        return result?.FinalAnswer;

And then inside the native function

        if(kernel.Data.TryGetValue("question", out var q) && q is Reference<string> qRef)
        {
            qRef.Inner = question;
        }
        if(kernel.Data.TryGetValue("stop", out var cts) && cts is CancellationTokenSource tokenSource)
        {
            tokenSource.Cancel();
        }
        // https://github.com/microsoft/semantic-kernel/blob/d1e81201b3bbafe72ba1bf7672ee2569ea062ca2/dotnet/src/Planners/Planners.OpenAI/Stepwise/FunctionCallingStepwisePlanner.cs#L355
        return "Thanks";

[sandeepvootoori]

@bjsmiley - What we ended up doing was utilizing manual function calling. We basically get the tool and then if we find that its a Native Function then we just stop running after that Function Execution.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[dmytrostruk]

@sandeepvootoori This functionality was achieved by introducing new type of filters for automatic function calling: https://github.com/microsoft/semantic-kernel/blob/0296329886eb2116a66e5362f2cc72b42ee30157/dotnet/samples/KernelSyntaxExamples/Example76_Filters.cs#L246-L283

Specifically, these lines of code should help: https://github.com/microsoft/semantic-kernel/blob/0296329886eb2116a66e5362f2cc72b42ee30157/dotnet/samples/KernelSyntaxExamples/Example76_Filters.cs#L272-L281

After next delegate is executed, context will contain function result. If you don't want this result to be sent to LLM and instead you just want to return it immediately, you can call context.Terminate = true. This setting will terminate function calling loop and will return function result.

This functionality is already merged to main branch and will be released next week. Hope this helps!

[gmantri]

We are using this functionality. Our use case is that for certain functions, we would want to stop the flow and execute the function manually. To stop the flow, we are using context.Terminate = true. It works great.

I do have some questions though:

1. How do we know in the main method that the flow was terminated (in other words, context.Terminate = true was set)?
2. How do we know the details of the function that was terminated?

[dmytrostruk]

• How do we know in the main method that the flow was terminated (in other words, context.Terminate = true was set)?
• How do we know the details of the function that was terminated?

@gmantri If you want this information just for general understanding of the flow, you can log this information. If you want this information to process programmatically and build some logic around it, you can use context.Kernel.Data property-bag, where you can put information about function that was terminated. Let me know if that works for you. Thanks!

[gmantri]

@dmytrostruk - Thanks for the reply. Using context.Kernel.Data will work for us.