.Net: Planner Prompt Examples Need To Be Customizable

[drewbrown27]

Describe the bug The examples injected into the planner prompt is negatively impacting the planners choices. We have removed the time plugin on our semantic kernel application because it was causing issues, yet the planner still selects 'TimePlugin.Time' even though it isn't listed in our list of functions.

To Reproduce Steps to reproduce the behavior:

1. Remove the Time Plugin and ask time related questions.
2. (Example) You can remove the Time examples from the planner prompt and it will not return the time function (correctly so)

Expected behavior The examples for the Planner Prompt should be either customizable or based off which functions you have imported. The examples sway the decision making of the LLM far too much to include functions in the examples that are not present in the actual function list.

Prompt A planner takes a list of functions, a goal, and chooses which function to use.
For each function the list includes details about the input parameters.
[START OF EXAMPLES]

[EXAMPLE]

• List of functions:
  // Read a file.
  FileIOPlugin.ReadAsync
  Parameter "path": Source file.
  // Write a file.
  FileIOPlugin.WriteAsync
  Parameter "path": Destination file. (default value: sample.txt)
  Parameter "content": File content.
  // Get the current time.
  TimePlugin.Time
  No parameters.
  // Makes a POST request to a uri.
  HttpPlugin.PostAsync
  Parameter "body": The body of the request.
• End list of functions.
  Goal: create a file called "something.txt".
  {"plan":{
  "rationale": "the list contains a function that allows to create files",
  "function": "FileIOPlugin.WriteAsync",
  "parameters": {
  "path": "something.txt",
  "content": null
  }}}

  END-OF-PLAN

[EXAMPLE]

• List of functions:
  // Get the current time.
  TimePlugin.Time
  No parameters.
  // Write a file.
  FileIOPlugin.WriteAsync
  Parameter "path": Destination file. (default value: sample.txt)
  Parameter "content": File content.
  // Makes a POST request to a uri.
  HttpPlugin.PostAsync
  Parameter "body": The body of the request.
  // Read a file.
  FileIOPlugin.ReadAsync
  Parameter "path": Source file.
• End list of functions.
  Goal: tell me a joke.
  {"plan":{
  "rationale": "the list does not contain functions to tell jokes or something funny",
  "function": "",
  "parameters": {
  }}}

  END-OF-PLAN

Platform

• OS: Windows
• IDE: Visual Studio
• Language: C#

Additional context Add any other context about the problem here.

[matthewbolanos]

@teresaqhoang, this could be a good addition to the Handlebars planner you're creating after your finished with the MVP.

[teresaqhoang]

@drewbrown27 @matthewbolanos this will be customizable with the Handlebars Planner, since we have to explicitly register all operations as helpers in the Handlebars instance.

We'll only be registering kernel functions as helpers if they're explicitly registered by the users to the kernel. No plugins will be added as helpers by default.

Instead, we'll be baking in utilities and operations as default helpers that will be non-ambiguous with any registered kernel functions. You'll also have the option to opt out of registering or filtering these built-in system helpers, following the same pattern as defined by the Handlebars.Helpers library.

More work to come to filter the helpers SK is baking in as default, but you can track the work here.

You can read all about the different buckets of helpers here: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0023-handlebars-template-engine.md

[teresaqhoang]

Hey @drewbrown27, we're deprecating the older Action, Sequential, and Stepwise Planner in favor of the HandlebarsPlanner and FunctionCallingStepwisePlanner.

The HandlebarsPlanner doesn't register any plugins by default, only the ones that have been imported to the kernel by the user, but we do have baked in functionality that is limited to the helpers defined in KernelSystemHelpers. These are mostly utility operations that allow the planner to use loops and engineer the plan better, so it shouldn't cause conflicts.

I have a fast follow work item #3947 that will allow you to filter or omit these built-in helpers.

Try it out and re-open this issue if you're still seeing conflicts!