search

mscraftsman / generative-ai

Gemini AI SDK for .NET and ASP.NET Core enables developers to use Google's state-of-the-art generative AI models to build AI-powered features and applications.
https://mscraftsman.github.io/generative-ai/
Apache License 2.0
55 stars 11 forks source link

Error while copying content to a stream #25

Open sebastianbk opened 6 months ago

sebastianbk commented 6 months ago

First of all, thank you so much for creating this library! You've done an outstanding job and made it so much easier to get started with Gemini as a .NET developer. Amazing work!

I've done some extensive testing and I continuously experience the error. It's quite erratic and I can't really tell why the error is being thrown. It seems random to me.

Is there anything I can do to avoid this?

Thanks!

Error while copying content to a stream.
System.Net.Http.HttpRequestException: Error while copying content to a stream.
      ---> System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
      at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
      at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
      at System.Net.Http.Http2Connection.Http2Stream.CopyToAsync(HttpResponseMessage responseMessage, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
      at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken)
      at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
      --- End of inner exception stack trace ---
      at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
      at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
      at Mscc.GenerativeAI.GenerativeModel.GenerateContent(GenerateContentRequest request)
      at Mscc.GenerativeAI.GenerativeModel.GenerateContent(String prompt, GenerationConfig generationConfig, List`1 safetySettings, List`1 tools, ToolConfig toolConfig)
jochenkirstaetter commented 6 months ago

Hello @sebastianbk

Yes, that's an issue with .NET itself and how the streaming content is received from the Google server. There is quite some discussion in the .NET source code about that. The root cause is that there seems to be too much "chatter" from the HttpClient and therefore the server closes the connection. I actually commented the references in the source code:

// Ref: https://github.com/dotnet/runtime/issues/97128 - HttpIOException // https://github.com/grpc/grpc-dotnet/issues/2361#issuecomment-1895805167

Currently, a potential workaround might be to switch to the "SSE" mode as Google calls it. It's based on the Server-Sent Events concept although it uses the response format only.

See test case https://github.com/mscraftsman/generative-ai/blob/96a4602796a6e2a200b26f6824910f35601211c3/tests/Mscc.GenerativeAI/GoogleAi_GeminiPro_Should.cs#L449 on how to use this format.

Hope this helps!

Cheers, JoKi

jochenkirstaetter commented 6 months ago

@sebastianbk

After I upgraded to the latest .NET 8 SDK and runtime, I observed that the issue seems to be less likely. However, the .NET team targets primarily .NET 9 with additional HTTP response exceptions that would provide the details.

Cheers, JoKi