Closed drewbrown27 closed 8 months ago
@teresaqhoang, this could be a good addition to the Handlebars planner you're creating after your finished with the MVP.
@drewbrown27 @matthewbolanos this will be customizable with the Handlebars Planner, since we have to explicitly register all operations as helpers in the Handlebars instance.
We'll only be registering kernel functions as helpers if they're explicitly registered by the users to the kernel. No plugins will be added as helpers by default.
Instead, we'll be baking in utilities and operations as default helpers that will be non-ambiguous with any registered kernel functions. You'll also have the option to opt out of registering or filtering these built-in system helpers, following the same pattern as defined by the Handlebars.Helpers library.
More work to come to filter the helpers SK is baking in as default, but you can track the work here.
You can read all about the different buckets of helpers here: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0023-handlebars-template-engine.md
Hey @drewbrown27, we're deprecating the older Action, Sequential, and Stepwise Planner in favor of the HandlebarsPlanner and FunctionCallingStepwisePlanner.
The HandlebarsPlanner
doesn't register any plugins by default, only the ones that have been imported to the kernel by the user, but we do have baked in functionality that is limited to the helpers defined in KernelSystemHelpers. These are mostly utility operations that allow the planner to use loops and engineer the plan better, so it shouldn't cause conflicts.
I have a fast follow work item #3947 that will allow you to filter or omit these built-in helpers.
Try it out and re-open this issue if you're still seeing conflicts!
Describe the bug The examples injected into the planner prompt is negatively impacting the planners choices. We have removed the time plugin on our semantic kernel application because it was causing issues, yet the planner still selects 'TimePlugin.Time' even though it isn't listed in our list of functions.
To Reproduce Steps to reproduce the behavior:
Expected behavior The examples for the Planner Prompt should be either customizable or based off which functions you have imported. The examples sway the decision making of the LLM far too much to include functions in the examples that are not present in the actual function list.
Prompt A planner takes a list of functions, a goal, and chooses which function to use.
For each function the list includes details about the input parameters.
[START OF EXAMPLES]
[EXAMPLE]
// Read a file.
FileIOPlugin.ReadAsync
Parameter "path": Source file.
// Write a file.
FileIOPlugin.WriteAsync
Parameter "path": Destination file. (default value: sample.txt)
Parameter "content": File content.
// Get the current time.
TimePlugin.Time
No parameters.
// Makes a POST request to a uri.
HttpPlugin.PostAsync
Parameter "body": The body of the request.
Goal: create a file called "something.txt".
{"plan":{
"rationale": "the list contains a function that allows to create files",
"function": "FileIOPlugin.WriteAsync",
"parameters": {
"path": "something.txt",
"content": null
}}}
END-OF-PLAN
[EXAMPLE]
// Get the current time.
TimePlugin.Time
No parameters.
// Write a file.
FileIOPlugin.WriteAsync
Parameter "path": Destination file. (default value: sample.txt)
Parameter "content": File content.
// Makes a POST request to a uri.
HttpPlugin.PostAsync
Parameter "body": The body of the request.
// Read a file.
FileIOPlugin.ReadAsync
Parameter "path": Source file.
Goal: tell me a joke.
{"plan":{
"rationale": "the list does not contain functions to tell jokes or something funny",
"function": "",
"parameters": {
}}}
END-OF-PLAN
Platform
Additional context Add any other context about the problem here.