.Net ConversationSummary Plugin doesn't work for gpt-35-turbo-16k v0613

[buzahid]

I have two Azure OpenAI model deployments, gpt-35-turbo v0301 and gpt-35-turbo-16k v0613. The ConversationSummary plugin works for gpt v301 but not v0613.

To Reproduce This is the code I used for both models:

kernel.ImportPluginFromType<ConversationSummaryPlugin>();

string input = @"I'm a vegan in search of new recipes. 
I love spicy food! Can you give me a list of breakfast 
recipes that are vegan friendly?";

var result = await kernel.InvokeAsync(
    "ConversationSummaryPlugin", 
    "GetConversationActionItems", 
    new() {{ "input", input }});

Console.WriteLine(result);

Expected behavior Expected the output to be the same between both models.

Output Using v0613

{
    "actionItems": []
}

Same code using 0301:

{
    "actionItems": [
        {
            "owner": "",
            "actionItem": "Provide a list of vegan-friendly spicy breakfast recipes",
            "dueDate": "",
            "status": "Open",
            "notes": ""
        }
    ]
}

Platform

• Windows
• VS Code
• C#, Python
• Microsoft.SemanticKernel.Plugins.Core, Version=1.1.0.0 (Also on Version=1.2.0.0)

[matthewbolanos]

@buzahid, it looks like this is an issue with the underlying prompt; we likely need to do some prompt engineering to make it work for the latest model. Would you be able to do some experimentation to see what the "correct" prompt should be for the plugin?

[JasonHaley]

I'm having the same issue. I can help experiment with different prompts if you still want to do that.

One thing to note is, I ran into it while going through the new Microsoft learn module on Semantic Kernel: https://learn.microsoft.com/en-us/training/paths/develop-ai-agents-azure-open-ai-semantic-kernel-sdk/

I also see an error if I turn on trace logging (but that may be expected since the resultValue is

{{ "actionItems": [] }}

func4f772a97653e470ebba2390cfb5168e6[0] Function result: Failed to serialize result value to Json System.NotSupportedException: Runtime type 'Microsoft.SemanticKernel.Connectors.OpenAI.OpenAIChatMessageContent' is not supported by polymorphic type 'Microsoft.SemanticKernel.KernelContent'. Path: $. ---> System.NotSupportedException: Runtime type 'Microsoft.SemanticKernel.Connectors.OpenAI.OpenAIChatMessageContent' is not supported by polymorphic type 'Microsoft.SemanticKernel.KernelContent'. at System.Text.Json.ThrowHelper.ThrowNotSupportedException_RuntimeTypeNotSupported(Type baseType, Type runtimeType) at System.Text.Json.Serialization.Metadata.PolymorphicTypeResolver.TryGetDerivedJsonTypeInfo(Type runtimeType, JsonTypeInfo& jsonTypeInfo, Object& typeDiscriminator) at System.Text.Json.Serialization.JsonConverter.ResolvePolymorphicConverter(Object value, JsonTypeInfo jsonTypeInfo, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) --- End of inner exception stack trace --- at System.Text.Json.ThrowHelper.ThrowNotSupportedException(WriteStack& state, NotSupportedException ex) at System.Text.Json.Serialization.JsonConverter1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValu

[kduenke]

I experienced the same behavior (i.e. the empty actionItems array) while I was following the same Learn content already mentioned. Since I'm new to SK, it really confused me and threw me for a loop. I didn't know if I had done something wrong or otherwise. I was never able to make it work, not on gpt-35-turbo v0301 or gpt-35-turbo-16k v0613. I'm using Microsoft.SemanticKernel.Plugins.Core 1.10.0-alpha.

Can we fix either the plug-in or the Learn content?

[npv00]

confirming @buzahid above - i had a project and kernel.InvokeAsync returning empty action items - created new deployment using model gpt-35-turbo which was (at the time of this writing) ver 0301 and my code worked. I suspect if you do same, you code above will work as well.

[odonyde]

The issue is still present using current prerelease version 1.13.0-alpha of NuGet package Microsoft.SemanticKernel.Plugins.Core and current default version 0613 of model gpt-35-turbo-16k.

I have raised an issue for learning module Create plugins for semantic kernel on Microsoft Learn, suggesting to substitute function GetConversationActionItems with function GetConversationTopics for the purpose of the exercise to not throw off students by always receiving an empty actionItems array.

[zeecorleone]

I encountered the same problem. When I tried to look into the root cause, I found the problem actually was related to the prompt.

The prompt was more used to understand the chat style conversation, unlike the one mentioned in the problem description: I'm a vegan in search of new recipes. I love spicy food! Can you give me a list of breakfast recipes that are vegan friendly? (Which is btw also being used in one if the MS learning path code samples)

I have initiated this PR with the improvement in prompt. it became a bit tricky, as it also tried to extract action item from simple statements. For example, for given input I'm a superhero. I go out for super walk. It started extracting going out for walk as action item. Hence I had to add "Guidelines" section in prompt to counter that