OpenAI library doesn't work with Gemini's OpenAI compat endpoint

[stephentoub]

Service

OpenAI

Describe the bug

Google blogged about how they now expose an OpenAI-compatible endpoint for gemini: https://developers.googleblog.com/en/gemini-is-now-accessible-from-the-openai-library/

but the .NET OpenAI library doesn't work with it. Gemini is sending back a role of "model" as part of the response, but the OpenAI library is trying to validate that any role is one of only five hardcoded values, and anything else causes the request to fail: https://github.com/openai/openai-dotnet/blob/539172fc699d29c3a812c848b07dc7e67fc7630f/src/Generated/Models/ChatMessageRole.Serialization.cs#L21-L29

Steps to reproduce

Run the .NET equivalent to the code in that blog post:

using OpenAI;
using OpenAI.Chat;

var client = new OpenAIClient(new("gemini_api_key"), new()
{
    Endpoint = new("https://generativelanguage.googleapis.com/v1beta/"),
}).GetChatClient("gemini-1.5-flash");

var response = await client.CompleteChatAsync(
[
    ChatMessage.CreateSystemMessage("You are a helpful AI assistant."),
    ChatMessage.CreateUserMessage("Explain to me how AI works"),
]);

Console.WriteLine(response.Value.Content[0].Text);

It fails with:

Unhandled exception. System.ArgumentOutOfRangeException: Unknown ChatMessageRole value. (Parameter 'value')
Actual value was model.
   at OpenAI.Chat.ChatMessageRoleExtensions.ToChatMessageRole(String value)
   at OpenAI.Chat.InternalChatCompletionResponseMessage.DeserializeInternalChatCompletionResponseMessage(JsonElement element, ModelReaderWriterOptions options)
   at OpenAI.Chat.InternalCreateChatCompletionResponseChoice.DeserializeInternalCreateChatCompletionResponseChoice(JsonElement element, ModelReaderWriterOptions options)
   at OpenAI.Chat.ChatCompletion.DeserializeChatCompletion(JsonElement element, ModelReaderWriterOptions options)
   at OpenAI.Chat.ChatCompletion.FromResponse(PipelineResponse response)
   at OpenAI.Chat.ChatClient.CompleteChatAsync(IEnumerable`1 messages, ChatCompletionOptions options, CancellationToken cancellationToken)
   at OpenAI.Chat.ChatClient.CompleteChatAsync(ChatMessage[] messages)
   at Program.<Main>$(String[] args)

Code snippets

OS

any

.NET version

.NET 8

Library version

2.1.0-beta.2

[cosmez]

after adding the new role to the Serializer seems to be working fine. i assume this library is generated automatically. but i couldnt find instructions on how to fix the generation code. havent tested anything complex, but this patch seems to work:

--- a/src/Generated/Models/ChatMessageRole.Serialization.cs
+++ b/src/Generated/Models/ChatMessageRole.Serialization.cs
@@ -15,6 +15,7 @@ namespace OpenAI.Chat
             ChatMessageRole.Assistant => "assistant",
             ChatMessageRole.Tool => "tool",
             ChatMessageRole.Function => "function",
+            ChatMessageRole.Model => "model",
             _ => throw new ArgumentOutOfRangeException(nameof(value), value, "Unknown ChatMessageRole value.")
         };

@@ -23,6 +24,7 @@ namespace OpenAI.Chat
             if (StringComparer.OrdinalIgnoreCase.Equals(value, "system")) return ChatMessageRole.System;
             if (StringComparer.OrdinalIgnoreCase.Equals(value, "user")) return ChatMessageRole.User;
             if (StringComparer.OrdinalIgnoreCase.Equals(value, "assistant")) return ChatMessageRole.Assistant;
+            if (StringComparer.OrdinalIgnoreCase.Equals(value, "model")) return ChatMessageRole.Model;
             if (StringComparer.OrdinalIgnoreCase.Equals(value, "tool")) return ChatMessageRole.Tool;
             if (StringComparer.OrdinalIgnoreCase.Equals(value, "function")) return ChatMessageRole.Function;
             throw new ArgumentOutOfRangeException(nameof(value), value, "Unknown ChatMessageRole value.");

[jochenkirstaetter]

Hi, I ran into the same issue after a Googler pointed me to a Colab gist using Python.

https://discuss.ai.google.dev/t/new-endpoints-chat-embedding-and-openai/49954/6 https://colab.research.google.com/gist/Gunand3043/a9a0ec90d13eda6905c5abb2b00177d1/openai_compa.ipynb

The Gemini API returns a role of model which should be mapped to either Assistant or to a newly created enum value Model. I'd prefer the former. But I assume that the solution might be to extend the enumeration ChatMessageRole accordingly because of the generated code.

Anyone?

[jochenkirstaetter]

And here's the source code I used to reproduce this issue.

using OpenAI;
using OpenAI.Chat;
using System.ClientModel;

var apiKey = Environment.GetEnvironmentVariable("GOOGLE_API_KEY");
var model = "gemini-1.5-flash";
var prompt = "Explain to me how AI works.";

OpenAIClientOptions options = new() { Endpoint = new Uri("https://generativelanguage.googleapis.com/v1beta/") };
ChatClient client = new(model, new ApiKeyCredential(apiKey), options);
ChatCompletion completion = client.CompleteChat(prompt);
Console.WriteLine($"[ASSISTANT]: {completion.Content[0].Text}");

[jochenkirstaetter]

Maybe as an idea, here's the mapping I implemented in my Gemini client for Microsoft.Extensions.AI in order to map the returned roles to their ChatRole enumeration.

        private static ChatRole ToAbstractionRole(string? role)
        {
            if (string.IsNullOrEmpty(role)) return new ChatRole("unknown");

            return role switch
            {
                Role.User => ChatRole.User,
                Role.Model => ChatRole.Assistant,
                Role.System => ChatRole.System,
                Role.Function => ChatRole.Tool,
                _ => new ChatRole(role)
            };
        }

The incoming value of role parameter is one of the currently possible values coming from the Gemini API.

[jochenkirstaetter]

Hello

However, giving it a second thought. The Gemini API claims to be compatible to the OpenAI library. Given the current situation I would rather say that the returned value of role is wrong, and should be assistant instead of model.

Kind regards, JoKi

[cosmez]

the api will be compatible but a lot of applications won't work if the model role is needed.

[jochenkirstaetter]

You are probably right. However, this is a newly introduced value by Gemini only. All the others currently supported are working as expected. And as mentioned, it's the Gemini endpoint that proclaimed to be compatible with OpenAI. Not the other way around...

Just my thoughts on that.