.Net: Function created with CreateSemanticFunction() not used in planner

[MikeYeager]

Describe the bug I recently updated a console project I use for testing from SK preview 14 to SK preview 19. It broke a bunch of stuff, which I fixed, but I'm having a new problem now. I load a few skills from disk, then call kernel.CreateSemanticFunction() using the stock "write in the style of Shakespeare" prompt. Then I call the planner. It used to rewrite the output in the style of Shakespeare, but it no longer includes this step. The documentation I found here https://learn.microsoft.com/en-us/semantic-kernel/get-started/quick-start-guide/using-the-planner?tabs=Csharp shows essentially the same steps. Is there a new step I'm missing?

To Reproduce

    //Load up the skills the planner will use
    var skillsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "skills");
    kernel.ImportSemanticSkillFromDirectory(skillsDirectory, "SummarizeSkill");
    kernel.ImportSemanticSkillFromDirectory(skillsDirectory, "WriterSkill");

    //Define a skill on the fly
    string skPrompt = """
        {{$input}}

        Rewrite the above in the style of Shakespeare.
    """;

    kernel.CreateSemanticFunction(skPrompt, "shakespeare", "ShakespeareSkill", maxTokens: 2000, temperature: 0.2, topP: 0.5);

    var goal = "Tomorrow is Valentine's day. I need to come up with a few date ideas.\r\nShe likes Shakespeare so write using his style. E-mail these ideas to my significant other.";
    var plannerConfig = new SequentialPlannerConfig { RelevancyThreshold = 0.5 };
    var planner = new SequentialPlanner(kernel, plannerConfig);
    var plan = await planner.CreatePlanAsync(goal);     //This line connects to Azure

    var i = 1;
    foreach (var step in plan.Steps)
        Console.WriteLine($"Step {i++}: {step.SkillName} {step.Description}");
    var results = await kernel.RunAsync(plan); //This line connects to Azure
    Console.WriteLine(results);

The output is: Step 1: WriterSkill Given a goal or topic description generate a list of ideas Step 2: WriterSkill Generate an acronym for the given concept or phrase Step 3: WriterSkill Given a request to generate an acronym from a string, generate an acronym and provide the acronym explanation. Step 4: WriterSkill Turn a scenario into a short and entertaining poem. Step 5: WriterSkill Summarize given text in two sentences or less Step 6: WriterSkill Turn bullet points into an email to someone, using a polite tone

Dear My Significant Other,

I hope you're doing well. I wanted to suggest some activities that we can do together to make our relationship even more special.

We could take a romantic walk in the park, have a picnic in the backyard, cook a romantic dinner together, enjoy a romantic movie night, take a cooking class together, visit a local art gallery, go for a bike ride, enjoy a romantic spa day, or take a dance class together. We could even go stargazing for a truly magical experience.

I'm looking forward to spending quality time with you and creating memories that will last a lifetime.

Thanks, Me

[nacharya1]

@lemillermicrosoft can you investigate this and see why this is occuring.

[lemillermicrosoft]

What model are you using?

You have a non-zero temperature, so results will definitely not be deterministic.

Additionally, there is no description for the function, possibly making it difficult for the model to infer correct usage.

When I ran it using gpt-4:

Step 1: WriterSkill Given a goal or topic description generate a list of ideas
Step 2: ShakespeareSkill Generic function, unknown purpose
Step 3: WriterSkill Turn bullet points into an email to someone, using a polite tone

Closing for now as no-repro, please re-open as appropriate.

[MikeYeager]

@lemillermicrosoft I'm using text-davinci-003. I literally updated from preview 0.14 to 0.19, fixed the issues that arose from things being moved around in SK and it stopped working. Adding a description seemed like a good idea (though it wasn't required before the update). But when I added a description to the CreateSemanticFunction() call: description: "Rewrites text in the style of Shakespeare", it started throwing an exception.

[lemillermicrosoft]

@MikeYeager can you include the text of the message? It will show the result of the LLM call and help better diagnose what is wrong. I'm trying text-davinci-003 right now and unable to get a repro with latest main (maybe we fixed something recently).

[MikeYeager]

@lemillermicrosoft Invalid plan: Failed to parse plan xml string: '

<function.ShakespeareSkill.shakespeare input="Tomorrow is Valentine's day. I need to come up with a few date ideas." setContextVariable="SHAKESPEARE_STYLE"/>
<!-- Generate a list of ideas -->
<function.WriterSkill.Brainstorm input="$SHAKESPEARE_STYLE" setContextVariable="IDEAS"/>
<!-- Turn bullet points into an email to someone, using a polite tone -->
<function.WriterSkill.EmailTo to="My Significant Other" input="$IDEAS" sender="Me" appendToResult="RESULT__EMAIL"/>'

[MikeYeager]

`Invalid plan: Failed to parse plan xml string: '

<function.ShakespeareSkill.shakespeare input="Tomorrow is Valentine's day. I need to come up with a few date ideas." setContextVariable="SHAKESPEARE_STYLE"/>
<!-- Generate a list of ideas -->
<function.WriterSkill.Brainstorm input="$SHAKESPEARE_STYLE" setContextVariable="IDEAS"/>
<!-- Turn bullet points into an email to someone, using a polite tone -->
<function.WriterSkill.EmailTo to="My Significant Other" input="$IDEAS" sender="Me" appendToResult="RESULT__EMAIL"/>'`

[MikeYeager]

Looks like it's objecting to the tag <plan>

[lemillermicrosoft]

Yeah, odd, it's not including the final </plan> tag and that is why parsing is failing. We could probably make this more robust, but can't commit to making it perfect with text-davinci-003. Will re-open for now, thanks for the details.