Closed Cotspheer closed 7 months ago
I have encountered the same issue. If you look at CreatePlanCoreAsync in dotnet/src/Planners/Planners.Handlebars/Handlebars /HandlebarsPlanner.cs, the detection for throwing this exception merely checks for presence of the error code anywhere in generated plan:
// Check if plan could not be created due to insufficient functions
if (completionResults.Content is not null && completionResults.Content.Contains(InsufficientFunctionsError))
{
var functionNames = availableFunctions.ToList().Select(func => $"{func.PluginName}{this._templateFactory.NameDelimiter}{func.Name}");
throw new KernelException($"[{HandlebarsPlannerErrorCodes.InsufficientFunctionsForGoal}] Unable to create plan for goal with available functions.\nGoal: {goal}\nAvailable Functions: {string.Join(", ", functionNames)}\nPlanner output:\n{completionResults}");
}
By instructing template to freely use conditionals, most of my plans end up with something similar to this:
{{#if query}}
{{json query}}
{{else}}
{{json "Additional helpers may be required"}}
{{/if}}
Perhaps prompt needs to be modified to make it clear to only output the error text unconditionally if plan cannot be constructed at all? As it is, as far as I can see handlebars planner is not really usable - using simple text check for the error trips me up in more than 75% of attempts to make a new plan dynamically.
This should be possible to fix with prompt engineering once we have partial support to override the template. We'll follow back up with this issue to see if it resolves your issue once it's done.
Hey @Cotspheer, @AieatAssam,
I agree that this error is more restrictive than we intended when we implemented it and have decided to remove this specific handler altogether. Instead, if the planner doesn't have enough functions to fulfill the plan, it should not return a Handlebars template at all and just return a sensical error string.
In your scenarios, the model should schedule an error message if the conditionals are not met.
I have the work in draft here: https://github.com/microsoft/semantic-kernel/pull/4983, will work on merging it once the changes to add more flexibility to the planner goes in: https://github.com/microsoft/semantic-kernel/pull/4918
I also considered adding a custom helper that could throw an error message, but I felt like that could be easily abused by the model — just wanted to bring this up in case you wanted to try it in your scenarios.
registerHelper("reportInsufficientFunctionsError", static (Context context, Arguments arguments) =>
{
throw new KernelException(HandlebarsPlanner.InsufficientFunctionsError);
});
Removed this error case altogether for being too unnecessarily restrictive: https://github.com/microsoft/semantic-kernel/pull/4983
Describe the bug As soon as there is a boolean-function as a plugin available like 'areLawsAvailable' the planner starts to create 'Additional helpers may be required' branches which then lead to exceptions. Currently every second attempt results in an exception which is a challenging user experience. The plan is perfectly fine and could be executed. Trying to inject things like "Never ever include 'Additional helpers may be required'." did not help.
To Reproduce Steps to reproduce the behavior:
Expected behavior Please remove this instruction from the template or give us a way to handle that properly. For example if the template should include such a part. Or let us customize the template.
Platform
Additional context
The generated plan:
The error response from my API:
The prompt used (only relevant parts as I can't disclose everything):