HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' when enabling `stream_options`

[jayendranarumugam]

I was using this reference code to calculate the openai streaming tokens. This was working all good.

                var originalStream = await responseContext.ProxyResponse.Content.ReadAsStreamAsync();
                var stringBuilder = new StringBuilder();

                // Buffer for reading chunks
                byte[] buffer = new byte[8192];
                int bytesRead;

                // Read, inspect, and write the data in chunks - this is especially needed for streaming content
                while ((bytesRead = await originalStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
                {
                    // Convert the chunk to a string for inspection
                    var chunk = Encoding.UTF8.GetString(buffer, 0, bytesRead);

                    stringBuilder.Append(chunk);

                    // Write the unmodified chunk back to the response
                    await responseContext.HttpContext.Response.Body.WriteAsync(buffer, 0, bytesRead);
                }

However when i start using the new feature "stream_options": {"include_usage": true} which might only work for text prompt and not give the usage details for image inputs image_url due to some unknown limitation or a bug. Hence we have to use our own logic above for calculating the usage tokens.

Coming to the main issue. The above code is throwing The HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' (0x2). (HttpProtocolError)

Error Stack Trace:

Exception thrown: 'System.Net.Http.HttpProtocolException' in System.Private.CoreLib.dll
warn: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      ResponseHeaders: The destination returned a response that cannot be proxied back to the client.
      System.Net.Http.HttpProtocolException: The HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' (0x2). (HttpProtocolError)
         at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
         at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
         at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
         at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)

Exception thrown: 'System.InvalidOperationException' in System.Private.CoreLib.dll
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HN7CT37HO2VS", Request id "0HN7CT37HO2VS:00000001": An unhandled exception was thrown by the application.
      System.InvalidOperationException: The response cannot be cleared, it has already started sending.
         at Microsoft.AspNetCore.Http.ResponseExtensions.Clear(HttpResponse response)
         at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)
         at Yarp.ReverseProxy.Forwarder.ForwarderMiddleware.Invoke(HttpContext context)
         at Yarp.ReverseProxy.Health.PassiveHealthCheckMiddleware.Invoke(HttpContext context)

[azureholic]

I don't think those stream options are supported by the Azure OpenAI API yet, or am I missing something?

[jayendranarumugam]

It is supported now. I can able to get the desired results using 2024-06-01 version. The only downside it its not working for image_url type alone, rest all working fine

[azureholic]

I've been waiting for this feature for a while now and keeping my eye on the docs and versions. There is no documentation on stream-options.

https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#data-plane-inference

are you sure you are calling an Azure OpenAI endpoint and not OpenAI itself?

[jayendranarumugam]

Yes I tried azure openai not openai. Its available there officially from the 2024-09-01-preview but its unofficially working even from 2024-06-01 :)

[azureholic]

thanks! I'll have a look at it