.Net: Sequential planner uses unknown steps

[rosieks]

Describe the bug Exception during creating plan:

PlanningException: Invalid plan: Failed to find function 'XML.ExtractValue' in skill 'XML'.

To Reproduce I get this exception while running the following code:

var kernel = new KernelBuilder()
    .WithAzureChatCompletionService("chat", "...", "...")
    .Build();

var casePlugin = await kernel.ImportChatGptPluginSkillFromUrlAsync("CasePlugin", new Uri("..."));

var planner = new SequentialPlanner(kernel);
var question = "Get me a list of cases and present names as a list. Do not use HTML.";
var plan = await planner.CreatePlanAsync(question);
var result = await plan.InvokeAsync();

Available functions:

CasePlugin.CreateCase:
  description: Creates new case
  inputs:
    - server_url: server-url (default value: ...)
  - payload: The case to be created (required)
  - content_type: Content type of REST API request body.

CasePlugin.GetAll:
  description: Get all cases
  inputs:
    - server_url: server-url (default value: ...)

Created plan:

<plan>
    <!-- Get all cases -->
    <function.CasePlugin.GetAll server_url="my-url" setContextVariable="CASES"/>
    <!-- Extract case names from the response -->
    <function.XML.ExtractValue input="$CASES" xpath="//case/name" setContextVariable="CASE_NAMES"/>
    <!-- Format case names as a list -->
    <function.String.Join input="$CASE_NAMES" delimiter=", " setContextVariable="CASE_NAMES_LIST"/>
    <!-- Output case names list -->
    <function.LogMessage message="Case names: $CASE_NAMES_LIST" />
</plan>

Expected behavior There is no exception while calling CreatePlanAsync

Screenshots N/A

Platform

• OS: Windows 11
• IDE: Visual Studio 2022
• Language: C#
• Source: 0.18.230725.3-preview

Additional context N/A

[evchaki]

@rosieks - thank you for bringing this up. We are looking into this bug.

[glahaye]

@teresaqhoang May or may not be related to changes you made recently. Just putting your name here for your visibility.

[teresaqhoang]

Hey @rosieks, yep, this is expected behavior. We error here to highlight functions that the model thinks could be useful to fulfil the plan but are not yet available in its context for use. We're also actively working on reducing these kinds of hallucinations from the model in the first place.

If you want the plan to error on the missing functions exception, make sure you don't have AllowMissingFunctions set to true in the config. It's false by default, so you don't need to explicitly call the planner with a config, but here's an example:

 new SequentialPlanner(
                    this.Kernel,
                    new SequentialPlannerConfig
                    {
                        // Throw an error on missing functions
                        AllowMissingFunctions = false
                    }
                )

If you want to bypass this missing functions error, see this PR here for a potential workaround: https://github.com/microsoft/chat-copilot/pull/87. Essentially, I set the AllowMissingFunctions option to true in SequentialPlannerConfig, allowing the planner to parse missing functions as no-op steps, and then filter out the no-op steps from the plan result.

 new SequentialPlanner(
                    this.Kernel,
                    new SequentialPlannerConfig
                    {
                        // Allow plan to be created with missing functions
                        AllowMissingFunctions = true
                    }
                )

[lemillermicrosoft]

Confirming @teresaqhoang response. This is the by design behavior. Please re-open or file a new issue if there are additional concerns.

As far as the model using invalid functions -- I'd recommend adding functions that it suggests or changing your query to better match your available functions. (i.e. don't say 'present the names as a list' if you don't have any skills that do presentation). You could create a semantic function that takes and input and does an action on it, too, to fill this gap.