.Net: The Gemini connector throws exceptions in parallel tool calls

[TaoChenOSU]

Describe the bug This is related to https://github.com/microsoft/semantic-kernel/issues/6281.

Using the Gemini connector results in exceptions. It still invokes the function multiple times. However, instead of returning a poem at the end, it throws an exception:

Unhandled exception. System.NotSupportedException: Runtime type 'Microsoft.SemanticKernel.Connectors.Google.GeminiChatMessageContent' is not supported by polymorphic type 'Microsoft.SemanticKernel.KernelContent'. Path: $.
 ---> System.NotSupportedException: Runtime type 'Microsoft.SemanticKernel.Connectors.Google.GeminiChatMessageContent' is not supported by polymorphic type 'Microsoft.SemanticKernel.KernelContent'.
   at System.Text.Json.ThrowHelper.ThrowNotSupportedException_RuntimeTypeNotSupported(Type baseType, Type runtimeType)
   at System.Text.Json.Serialization.Metadata.PolymorphicTypeResolver.TryGetDerivedJsonTypeInfo(Type runtimeType, JsonTypeInfo& jsonTypeInfo, Object& typeDiscriminator)
   at System.Text.Json.Serialization.JsonConverter.ResolvePolymorphicConverter(Object value, JsonTypeInfo jsonTypeInfo, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ThrowNotSupportedException(WriteStack& state, NotSupportedException ex)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed)
   at System.Text.Json.JsonSerializer.WriteNode[TValue](TValue& value, JsonTypeInfo`1 jsonTypeInfo)
   at System.Text.Json.JsonSerializer.SerializeToNode[TValue](TValue value, JsonSerializerOptions options)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiPart.FunctionResponsePart.FunctionResponseEntity..ctor(Object response)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiRequest.CreateGeminiParts(ChatMessageContent content)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiRequest.CreateGeminiContentFromChatMessage(ChatMessageContent message)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiRequest.AddChatMessage(ChatMessageContent message)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiChatCompletionClient.AddToolResponseMessage(ChatHistory chat, GeminiRequest request, GeminiFunctionToolCall tool, FunctionResult functionResponse, String errorMessage)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiChatCompletionClient.ProcessSingleToolCallAsync(ChatCompletionState state, GeminiFunctionToolCall toolCall, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiChatCompletionClient.ProcessFunctionsAsync(ChatCompletionState state, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Connectors.Google.Core.GeminiChatCompletionClient.GenerateChatMessageAsync(ChatHistory chatHistory, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.GetChatMessageContentAsync(IChatCompletionService chatCompletionService, String prompt, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeCoreAsync(Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.KernelFunction.<>c__DisplayClass21_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.SemanticKernel.Kernel.InvokeFilterOrFunctionAsync(NonNullCollection`1 functionFilters, Func`2 functionCallback, FunctionInvocationContext context, Int32 index)
   at Program.FirstFunctionFilter.OnFunctionInvocationAsync(FunctionInvocationContext context, Func`2 next) in C:\Users\taochen\Projects\ScratchPad\Program.cs:line 67
   at Microsoft.SemanticKernel.Kernel.InvokeFilterOrFunctionAsync(NonNullCollection`1 functionFilters, Func`2 functionCallback, FunctionInvocationContext context, Int32 index)
   at Microsoft.SemanticKernel.Kernel.OnFunctionInvocationAsync(KernelFunction function, KernelArguments arguments, FunctionResult functionResult, Func`2 functionCallback)
   at Microsoft.SemanticKernel.KernelFunction.InvokeAsync(Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Kernel.InvokeAsync[TResult](String pluginName, String functionName, KernelArguments arguments, CancellationToken cancellationToken)
   at Program.Main() in C:\Users\taochen\Projects\ScratchPad\Program.cs:line 29
   at Program.<Main>()

To Reproduce Steps to reproduce the behavior:

1. Follow the repro in: https://github.com/microsoft/semantic-kernel/issues/6281
2. Use the Gemini connector instead of the OpenAI connector.
3. Use a Gemini setting: GeminiPromptExecutionSettings settings = new() { ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions };

Expected behavior Return a poem.

There won't be any exception and the code correctly returns a poem with auto invoked disabled. This is because the function will only be invoked once.

Platform

• OS: Windows
• IDE: VS Code
• Language: C#
• Source: main branch of repository

Additional context Tested with gemini-1.0-pro

[Krzysztof318]

Currently there no simple way to fix this. Because it requires adding line [JsonDerivedType(typeof(GeminiChatMessageContent), typeDiscriminator: nameof(GeminiChatMessageContent))] to base class KernelContent or using special JsonTypeInfo but it is not supported on net standard.

But don't worry when function abstraction will be finished, probably classes GeminiChatMessageContent and StreamingGeminiChatMessageContent will be removed.