[.Net][Bug]: Image message fails with OpenAI

CyborTronik commented 4 months ago

Describe the bug

Getting invalid message type when using Image message with latest OpenAI (it supports image starting version 4)

Steps to reproduce

Created a C# app with AutoGen and OpenAI version 0.0.16. Try to send an Image message along with a text message.

For the piece of code like:

var image = await File.ReadAllBytesAsync(imagePath);
            var imageMessage = new ImageMessage(Role.Assistant, BinaryData.FromBytes(image, "image/jpg"));
            var reply = await agent.SendAsync("[query message]", [imageMessage]);

Model Used

gpt-4o/gpt-4o-mini

Expected Behavior

No response

Screenshots and logs

System.ArgumentException: Invalid message type
   at AutoGen.OpenAI.OpenAIChatAgent.<>c.<CreateChatCompletionsOptions>b__14_0(IMessage m)
   at System.Linq.Enumerable.SelectListIterator`2[[AutoGen.Core.IMessage, AutoGen.Core, Version=0.0.16.0, Culture=neutral, PublicKeyToken=4487feedbaab40bf],[Azure.AI.OpenAI.ChatRequestMessage, Azure.AI.OpenAI, Version=1.0.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8]].MoveNext()
   at System.Linq.Enumerable.Any[ChatRequestMessage](IEnumerable`1 source, Func`2 predicate)
   at AutoGen.OpenAI.OpenAIChatAgent.CreateChatCompletionsOptions(GenerateReplyOptions options, IEnumerable`1 messages)
   at AutoGen.OpenAI.OpenAIChatAgent.GenerateReplyAsync(IEnumerable`1 messages, GenerateReplyOptions options, CancellationToken cancellationToken)
   at AutoGen.Core.AgentExtension.SendAsync(IAgent agent, IMessage message, IEnumerable`1 chatHistory, CancellationToken ct)
   at AutoGen.Core.AgentExtension.SendAsync(IAgent agent, String message, IEnumerable`1 chatHistory, CancellationToken ct)
   at [XXXXXX My Class1] (String imagePath) in C:\Users\trifa\source\repos\[XXXXXX My Class].cs:line 37
   at [XXXXXX My Class2].SetupResult(FileResult photo) in C:\Users\trifa\source\repos\[XXXXXX My Class2].cs:line 47
   at [XXXXXX].TakeAPhoto() in C:\Users\trifa\source\repos\[XXXXXX My Class2].cs:line 32
   at CommunityToolkit.Mvvm.Input.AsyncRelayCommand.AwaitAndThrowIfFailed(Task executionTask) in /_/src/CommunityToolkit.Mvvm/Input/AsyncRelayCommand.cs:line 351
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_0(Object state)
   at Android.App.SyncContext.<>c__DisplayClass2_0.<Post>b__0() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.App/SyncContext.cs:line 36
   at Java.Lang.Thread.RunnableImplementor.Run() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Java.Lang/Thread.cs:line 36
   at Java.Lang.IRunnableInvoker.n_Run(IntPtr jnienv, IntPtr native__this) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/net8.0/android-34/mcw/Java.Lang.IRunnable.cs:line 84
   at Android.Runtime.JNINativeWrapper.Wrap_JniMarshal_PP_V(_JniMarshal_PP_V callback, IntPtr jnienv, IntPtr klazz) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.Runtime/JNINativeWrapper.g.cs:line 26

Additional Information

No response

LittleLittleCloud commented 4 months ago

I try the Image_Chat_With_Agent tutorial with gpt-4o-mini and it's working. So could you share the code and full stack trace, the one you shared above is not complete?

CyborTronik commented 4 months ago

@LittleLittleCloud updated the stack trace and added few details

LittleLittleCloud commented 4 months ago

@CyborTronik Did you register the message connector?

LittleLittleCloud commented 3 months ago

Close due to no response, feel free to reopen it if the error still exists

microsoft / autogen