search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.17k stars 4.72k forks source link

[HTTP/2] Race between reading response and reset? #57617

Open JamesNK opened 3 years ago

JamesNK commented 3 years ago

The server is completing the response stream and then sending RST_STREAM frame.

The client should be able to read the response to the end without error. Instead, they are getting an error.

This is from a flakey test. This is the only time I've seen it fail.

 WriteMessageAfterDeadline
   Source: DeadlineTests.cs line 114
   Duration: 218 ms

  Message: 
System.IO.IOException : The request was aborted.
  ----> System.Net.Http.Http2ConnectionException : The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).

  Stack Trace: 
Http2Connection.ThrowRequestAborted(Exception innerException)
Http2Stream.CheckResponseBodyState()
Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
StreamPipeReader.<ReadAsync>g__Core|36_0(StreamPipeReader reader, CancellationTokenSource tokenSource, CancellationToken cancellationToken)
PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken) line 268
TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 72
MessageHelpers.AssertReadStreamMessageAsync[T](PipeReader pipeReader, String compressionEncoding, List`1 compressionProviders) line 102
TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 72
<<WriteMessageAfterDeadline>b__2>d.MoveNext() line 187
--- End of stack trace from previous location ---
TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout, String filePath, Int32 lineNumber) line 96
DeadlineTests.WriteMessageAfterDeadline() line 202
GenericAdapter`1.BlockUntilCompleted()
NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
AsyncToSyncAdapter.Await(Func`1 invoke)
TestMethodCommand.RunTestMethod(TestExecutionContext context)
TestMethodCommand.Execute(TestExecutionContext context)
<>c__DisplayClass1_0.<Execute>b__0()
BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
--Http2ConnectionException
Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
RttEstimator.OnPingAckReceived(Int64 payload, Http2Connection connection)
Http2Connection.ProcessPingFrame(FrameHeader frameHeader)
Http2Connection.ProcessIncomingFramesAsync()

  Standard Output: 
0.000s GrpcTestContext - Information: Starting DeadlineTests.WriteMessageAfterDeadline
0.001s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Discovering gRPC methods for FunctionalTestsWebsite.Infrastructure.DynamicService.
0.002s SERVER Grpc.AspNetCore.Server.Model.Internal.BinderServiceMethodProvider - Debug: Could not find bind method for FunctionalTestsWebsite.Infrastructure.DynamicService.
0.002s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Added gRPC method 'WriteUntilError' to service 'DynamicService'. Method type: 'ServerStreaming', route pattern: '/DynamicService/WriteUntilError'.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB1REBPPE4G" accepted.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB1REBPPE4G" started.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received SETTINGS frame for stream ID 0 with length 12 and flags NONE.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received HEADERS frame for stream ID 1 with length 90 and flags END_HEADERS.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received DATA frame for stream ID 1 with length 12 and flags NONE.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received DATA frame for stream ID 1 with length 0 and flags END_STREAM.
0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received SETTINGS frame for stream ID 0 with length 0 and flags ACK.
0.004s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request starting HTTP/2 POST http://127.0.0.1:50050/DynamicService/WriteUntilError application/grpc -
0.004s SERVER Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware - Debug: Request matched endpoint 'gRPC - /DynamicService/WriteUntilError'
0.005s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executing endpoint 'gRPC - /DynamicService/WriteUntilError'
0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Request deadline timeout of 00:00:00.2000000 started.
0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Reading message.
0.005s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB1REBPPE4G", Request id "0HMB1REBPPE4G:00000001": started reading request body.
0.005s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB1REBPPE4G", Request id "0HMB1REBPPE4G:00000001": done reading request body.
0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Deserializing 7 byte message to 'Greet.HelloRequest'.
0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Received message.
0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending HEADERS frame for stream ID 1 with length 59 and flags END_HEADERS.
0.006s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Sending message.
0.006s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Serialized 'Greet.HelloReply' to 22 byte message.
0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending DATA frame for stream ID 1 with length 27 and flags NONE.
0.006s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Message sent.
0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received PING frame for stream ID 0 with length 8 and flags NONE.
0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending PING frame for stream ID 0 with length 8 and flags ACK.
0.215s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Request with timeout of 00:00:00.2000000 has exceeded its deadline.
0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending HEADERS frame for stream ID 1 with length 47 and flags END_STREAM, END_HEADERS.
0.216s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Resetting response stream with error code 0.
0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Trace id "0HMB1REBPPE4G:00000001": HTTP/2 stream error "NO_ERROR". A Reset is being sent to the stream.
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 stream was reset by the application with error code NO_ERROR.
0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending RST_STREAM frame for stream ID 1 with length 4 and flags 0x0.
0.217s SERVER Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets - Debug: Connection id "0HMB1REBPPE4G" reset.
0.217s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Connection id "0HMB1REBPPE4G" is closed. The last processed stream ID was 1.
0.217s SERVER Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets - Debug: Connection id "0HMB1REBPPE4G" sending FIN because: "The client closed the connection."
0.219s GrpcTestContext - Information: Finishing DeadlineTests.WriteMessageAfterDeadline

Important bit:

0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending HEADERS frame for stream ID 1 with length 47 and flags END_STREAM, END_HEADERS.
0.216s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Resetting response stream with error code 0.
0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Trace id "0HMB1REBPPE4G:00000001": HTTP/2 stream error "NO_ERROR". A Reset is being sent to the stream.
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 stream was reset by the application with error code NO_ERROR.
0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending RST_STREAM frame for stream ID 1 with length 4 and flags 0x0.

The server is sending END_STREAM flag, then sending RST_STREAM frame.

.NET SDK (reflecting any global.json):
 Version:   6.0.100-rc.1.21417.11
 Commit:    ca60baef37
ghost commented 3 years ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
The server is completing the response stream and then sending RST_STREAM frame. The client should be able to read the response to the end without error. Instead, they are getting an error. This is from a flakey test. This is the only time I've seen it fail. ```  WriteMessageAfterDeadline  Source: DeadlineTests.cs line 114  Duration: 218 ms Message:  System.IO.IOException : The request was aborted. ----> System.Net.Http.Http2ConnectionException : The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). Stack Trace:  Http2Connection.ThrowRequestAborted(Exception innerException) Http2Stream.CheckResponseBodyState() Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead) Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken) StreamPipeReader.g__Core|36_0(StreamPipeReader reader, CancellationTokenSource tokenSource, CancellationToken cancellationToken) PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken) line 268 TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 72 MessageHelpers.AssertReadStreamMessageAsync[T](PipeReader pipeReader, String compressionEncoding, List`1 compressionProviders) line 102 TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 72 <b__2>d.MoveNext() line 187 --- End of stack trace from previous location --- TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout, String filePath, Int32 lineNumber) line 96 DeadlineTests.WriteMessageAfterDeadline() line 202 GenericAdapter`1.BlockUntilCompleted() NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter) AsyncToSyncAdapter.Await(Func`1 invoke) TestMethodCommand.RunTestMethod(TestExecutionContext context) TestMethodCommand.Execute(TestExecutionContext context) <>c__DisplayClass1_0.b__0() BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) --Http2ConnectionException Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode) RttEstimator.OnPingAckReceived(Int64 payload, Http2Connection connection) Http2Connection.ProcessPingFrame(FrameHeader frameHeader) Http2Connection.ProcessIncomingFramesAsync() Standard Output:  0.000s GrpcTestContext - Information: Starting DeadlineTests.WriteMessageAfterDeadline 0.001s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Discovering gRPC methods for FunctionalTestsWebsite.Infrastructure.DynamicService. 0.002s SERVER Grpc.AspNetCore.Server.Model.Internal.BinderServiceMethodProvider - Debug: Could not find bind method for FunctionalTestsWebsite.Infrastructure.DynamicService. 0.002s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Added gRPC method 'WriteUntilError' to service 'DynamicService'. Method type: 'ServerStreaming', route pattern: '/DynamicService/WriteUntilError'. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB1REBPPE4G" accepted. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB1REBPPE4G" started. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received SETTINGS frame for stream ID 0 with length 12 and flags NONE. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received HEADERS frame for stream ID 1 with length 90 and flags END_HEADERS. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received DATA frame for stream ID 1 with length 12 and flags NONE. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received DATA frame for stream ID 1 with length 0 and flags END_STREAM. 0.004s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received SETTINGS frame for stream ID 0 with length 0 and flags ACK. 0.004s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request starting HTTP/2 POST http://127.0.0.1:50050/DynamicService/WriteUntilError application/grpc - 0.004s SERVER Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware - Debug: Request matched endpoint 'gRPC - /DynamicService/WriteUntilError' 0.005s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executing endpoint 'gRPC - /DynamicService/WriteUntilError' 0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Request deadline timeout of 00:00:00.2000000 started. 0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Reading message. 0.005s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB1REBPPE4G", Request id "0HMB1REBPPE4G:00000001": started reading request body. 0.005s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB1REBPPE4G", Request id "0HMB1REBPPE4G:00000001": done reading request body. 0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Deserializing 7 byte message to 'Greet.HelloRequest'. 0.005s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Received message. 0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending HEADERS frame for stream ID 1 with length 59 and flags END_HEADERS. 0.006s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Sending message. 0.006s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Serialized 'Greet.HelloReply' to 22 byte message. 0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending DATA frame for stream ID 1 with length 27 and flags NONE. 0.006s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Message sent. 0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" received PING frame for stream ID 0 with length 8 and flags NONE. 0.006s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending PING frame for stream ID 0 with length 8 and flags ACK. 0.215s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Request with timeout of 00:00:00.2000000 has exceeded its deadline. 0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending HEADERS frame for stream ID 1 with length 47 and flags END_STREAM, END_HEADERS. 0.216s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Resetting response stream with error code 0. 0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Trace id "0HMB1REBPPE4G:00000001": HTTP/2 stream error "NO_ERROR". A Reset is being sent to the stream. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 stream was reset by the application with error code NO_ERROR. 0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending RST_STREAM frame for stream ID 1 with length 4 and flags 0x0. 0.217s SERVER Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets - Debug: Connection id "0HMB1REBPPE4G" reset. 0.217s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Connection id "0HMB1REBPPE4G" is closed. The last processed stream ID was 1. 0.217s SERVER Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets - Debug: Connection id "0HMB1REBPPE4G" sending FIN because: "The client closed the connection." 0.219s GrpcTestContext - Information: Finishing DeadlineTests.WriteMessageAfterDeadline ``` Important bit: ``` 0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending HEADERS frame for stream ID 1 with length 47 and flags END_STREAM, END_HEADERS. 0.216s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Resetting response stream with error code 0. 0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Trace id "0HMB1REBPPE4G:00000001": HTTP/2 stream error "NO_ERROR". A Reset is being sent to the stream. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 stream was reset by the application with error code NO_ERROR. 0.216s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB1REBPPE4G" sending RST_STREAM frame for stream ID 1 with length 4 and flags 0x0. ``` The server is sending END_STREAM flag, then sending RST_STREAM frame. ``` .NET SDK (reflecting any global.json): Version: 6.0.100-rc.1.21417.11 Commit: ca60baef37 ```
Author: JamesNK
Assignees: -
Labels: `area-System.Net.Http`
Milestone: -
geoffkizer commented 3 years ago

We have logic to handle this exact case here: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs#L904

It sounds like you only see this once in a while? Then it's likely we have a race here. It's possible we've introduced this in 6.0, but it's also possible it's always existed and is just rare to see in practice.

If I'm reading the log above correctly, it looks like there is a small request body (12 bytes) and Kestrel has received the entire request body before sending any response. Correct?

We should investigate for 6.0.

JamesNK commented 3 years ago

It sounds like you only see this once in a while?

It's very rare. I've seen it once in .NET 6 (this time).

If I'm reading the log above correctly, it looks like there is a small request body (12 bytes) and Kestrel has received the entire request body before sending any response. Correct?

Yes. A normal POST then response (with trailers).

The only interesting bit is the server sends RST_STREAM after END_STREAM. The END_STREAM is on the trailers frame.

geoffkizer commented 3 years ago

Looking at the code I linked, this seems funny to me:

When this situation happens (RST_STREAM with NO_ERROR after END_STREAM), we check if the request stream is still being sent. If it is, we will set _requestBodyAbandoned = true, then set requestBodyCancellationSource = _requestBodyCancellationSource, and fall out of the lock statement, at which point we check if requestBodyCancellationSource != null and then cancel the request body.

But if the request body has already completed sending, then we don't do this and we will end up calling Cancel(), which will cancel the response, including discarding any buffered data. That seems bad. We should not call Cancel in this case.

geoffkizer commented 3 years ago

Actually, ignore my comment above. The logic I pointed at seems fine. Since we have already received END_STREAM on the response, we must still be sending the request stream; otherwise the stream would be fully completed and we'd just ignore the RST_STREAM (because the stream would no longer be in our active stream table).

(Edit: We should probably Assert that the request body hasn't completed yet, instead of testing for it; but that's not critical.)

The issue seems to be related to PING handling in the RttEstimator. We are throwing an Http2ConnectionException because we think there's something wrong with the PING ACK that Kestrel has sent -- either we aren't expecting it, or it has the wrong PING value.

@JamesNK Any chance you could add the ping value to the log info here?

@antonfirsov Any ideas?

JamesNK commented 3 years ago

Kestrel doesn't log the content of pings. I could add it to a custom Kestrel build and rerun the test, but I've only ever seen this once so it is unlikely to reproduce.

What I will do is enable capturing event source logging for this test. If I see it again then that info will be available.

geoffkizer commented 3 years ago

@antonfirsov -- we probably should have some log events associated with these failure cases so we can more easily determine what's going on here.

antonfirsov commented 3 years ago

Unfortunately, we are not logging the received ping content either (only sends), so it's very hard to reason about the error with the information we can get with our existing logging.

I assume this happened on the CI. @JamesNK any chance you can try running the test in a loop or do some other trick to get a local repro? If we could somehow get a deterministic repro, we could experiment with extended logging and hunt down the bug.

@geoffkizer I think we should improve our logging for RC2 (or even RC1 if still possible), so if we get similar reports we would be able to react easier. Will it be hard to get it through the backports?

karelz commented 3 years ago

Triage: If there is really issue, it is bad -- it may be in client or on server. So far it happened only once. Without repro we cannot take action :( We need more logs -- we should add logging and try to get it to 6.0 to have at least chance to root cause it next time it happens somewhere (e.g. in customer production).

JamesNK commented 3 years ago

I saw this again from another gRPC test:

```  UnaryMethodDeadlineExceeded(False)  Source: DeadlineTests.cs line 58  Duration: 329 ms Message:  System.Net.Http.HttpRequestException : An error occurred while sending the request. ----> System.IO.IOException : The request was aborted. ----> System.Net.Http.Http2ConnectionException : The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). Stack Trace:  Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) GrpcWebHandler.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) line 165 HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 72 DeadlineTests.UnaryMethodDeadlineExceeded(Boolean throwErrorOnCancellation) line 106 GenericAdapter`1.BlockUntilCompleted() NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter) AsyncToSyncAdapter.Await(Func`1 invoke) TestMethodCommand.RunTestMethod(TestExecutionContext context) TestMethodCommand.Execute(TestExecutionContext context) <>c__DisplayClass1_0.b__0() BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) --IOException Http2Connection.ThrowRequestAborted(Exception innerException) Http2Stream.CheckResponseBodyState() Http2Stream.TryEnsureHeaders() Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --Http2ConnectionException Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode) Http2Connection.ProcessSettingsFrame(FrameHeader frameHeader, Boolean initialFrame) Http2Connection.ProcessIncomingFramesAsync() Standard Output:  0.000s GrpcTestContext - Information: Starting DeadlineTests.UnaryMethodDeadlineExceeded(False) 0.000s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Discovering gRPC methods for FunctionalTestsWebsite.Infrastructure.DynamicService. 0.000s SERVER Grpc.AspNetCore.Server.Model.Internal.BinderServiceMethodProvider - Debug: Could not find bind method for FunctionalTestsWebsite.Infrastructure.DynamicService. 0.000s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Added gRPC method 'WaitUntilDeadline-False' to service 'DynamicService'. Method type: 'Unary', route pattern: '/DynamicService/WaitUntilDeadline-False'. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB50CP10NC7" accepted. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB50CP10NC7" started. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" received SETTINGS frame for stream ID 0 with length 12 and flags NONE. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK. 0.002s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0. 0.003s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" received HEADERS frame for stream ID 1 with length 102 and flags END_HEADERS. 0.003s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" received DATA frame for stream ID 1 with length 12 and flags NONE. 0.003s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" received SETTINGS frame for stream ID 0 with length 0 and flags ACK. 0.003s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" received DATA frame for stream ID 1 with length 0 and flags END_STREAM. 0.003s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request starting HTTP/2 POST http://127.0.0.1:50050/DynamicService/WaitUntilDeadline-False application/grpc-web - 0.003s SERVER Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware - Debug: Request matched endpoint 'gRPC - /DynamicService/WaitUntilDeadline-False' 0.003s SERVER Grpc.AspNetCore.Web.Internal.GrpcWebMiddleware - Debug: Detected gRPC-Web request from content-type 'application/grpc-web'. 0.003s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executing endpoint 'gRPC - /DynamicService/WaitUntilDeadline-False' 0.003s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Request deadline timeout of 00:00:00.3000000 started. 0.003s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Reading message. 0.003s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB50CP10NC7", Request id "0HMB50CP10NC7:00000001": started reading request body. 0.003s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB50CP10NC7", Request id "0HMB50CP10NC7:00000001": done reading request body. 0.003s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Deserializing 7 byte message to 'Greet.HelloRequest'. 0.003s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Received message. 0.323s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Request with timeout of 00:00:00.3000000 has exceeded its deadline. 0.323s SERVER Grpc.AspNetCore.Web.Internal.GrpcWebMiddleware - Debug: Sending gRPC-Web response with content-type 'application/grpc-web'. 0.323s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" sending HEADERS frame for stream ID 1 with length 114 and flags END_STREAM, END_HEADERS. 0.324s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Resetting response stream with error code 0. 0.324s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Trace id "0HMB50CP10NC7:00000001": HTTP/2 stream error "NO_ERROR". A Reset is being sent to the stream. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 stream was reset by the application with error code NO_ERROR. 0.324s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB50CP10NC7" sending RST_STREAM frame for stream ID 1 with length 4 and flags 0x0. 0.324s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Request deadline stopped. 0.324s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executed endpoint 'gRPC - /DynamicService/WaitUntilDeadline-False' 0.324s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request finished HTTP/2 POST http://127.0.0.1:50050/DynamicService/WaitUntilDeadline-False application/grpc-web - - 200 0 application/grpc-web 321.0352ms 0.325s SERVER Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets - Debug: Connection id "0HMB50CP10NC7" reset. 0.325s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Debug: Connection id "0HMB50CP10NC7" is closed. The last processed stream ID was 1. 0.325s SERVER Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets - Debug: Connection id "0HMB50CP10NC7" sending FIN because: "The client closed the connection." 0.326s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB50CP10NC7" stopped. 0.330s GrpcTestContext - Information: Finishing DeadlineTests.UnaryMethodDeadlineExceeded(False) ```

SDK info:

.NET SDK (reflecting any global.json):
 Version:   6.0.100-rc.1.21421.6
 Commit:    77703ed54e
JamesNK commented 3 years ago

I haven't been able to repo it by putting the test in a loop.

JamesNK commented 3 years ago

Saw it again. This time on the CI server. Also, this test has your HTTP event source logs:

``` Failed ClientStreaming_MultipleWritesAndRetries_Failure [51 ms] Error Message: Grpc.Core.RpcException : Status(StatusCode="Unknown", Detail="Exception was thrown by handler. IOException: The client reset the request stream.") Stack Trace: at Grpc.Net.Client.Internal.Retry.RetryCall`2.<>c__DisplayClass12_0.<b__0>d.MoveNext() in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.Net.Client/Internal/Retry/RetryCall.cs:line 294 --- End of stack trace from previous location --- at Grpc.Net.Client.Internal.Retry.RetryCall`2.DoClientStreamActionAsync(Func`2 action) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.Net.Client/Internal/Retry/RetryCall.cs:line 324 at Grpc.Net.Client.Internal.Retry.RetryCall`2.DoClientStreamActionAsync(Func`2 action) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.Net.Client/Internal/Retry/RetryCall.cs:line 332 at Grpc.Tests.Shared.TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout, String filePath, Int32 lineNumber) in /home/runner/work/grpc-dotnet/grpc-dotnet/test/Shared/TaskExtensions.cs:line 102 at Grpc.AspNetCore.FunctionalTests.Client.RetryTests.ClientStreaming_MultipleWritesAndRetries_Failure() in /home/runner/work/grpc-dotnet/grpc-dotnet/test/FunctionalTests/Client/RetryTests.cs:line 88 at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted() in nunit.framework.dll:token 0x6000d7a+0x7 at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter) in nunit.framework.dll:token 0x6000d33+0x1 at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke) in nunit.framework.dll:token 0x60003e4+0x34 at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context) in nunit.framework.dll:token 0x6000839+0x3b at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context) in nunit.framework.dll:token 0x6000838+0x1 at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.b__0() in nunit.framework.dll:token 0x6000e09+0x18 at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) in nunit.framework.dll:token 0x6000815+0x2 Standard Output Messages: 0.000s GrpcTestContext - Information: Starting RetryTests.ClientStreaming_MultipleWritesAndRetries_Failure 0.010s HttpEventSourceListener - Debug: Starting HttpEventSourceListener. 0.010s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Discovering gRPC methods for FunctionalTestsWebsite.Infrastructure.DynamicService. 0.010s SERVER Grpc.AspNetCore.Server.Model.Internal.BinderServiceMethodProvider - Debug: Could not find bind method for FunctionalTestsWebsite.Infrastructure.DynamicService. 0.010s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s SERVER Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder - Trace: Added gRPC method 'f641dcc8-16bf-4f33-ad7b-c61b5d971eab' to service 'DynamicService'. Method type: 'ClientStreaming', route pattern: '/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab'. 0.010s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.010s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.011s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.012s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.013s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.014s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.016s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.017s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s Grpc.Net.Client.Internal.RetryCall - Trace: Starting retry worker. 0.022s Grpc.Net.Client.Internal.RetryCall - Debug: Starting attempt 1. 0.022s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.022s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : PushStreamContent`2#22034886,.ctor, -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Associate : HttpRequestMessage#1950778,Content,HttpRequestMessage#1950778,PushStreamContent`2#22034886 -> 0.023s Grpc.Net.Client.Internal.GrpcCall - Debug: Starting gRPC call. Method type: 'ClientStreaming', URI: 'http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab'. 0.023s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestStart : http,127.0.0.1,50050,/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab,2,0,1 -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : HttpConnectionHandler#51888618,.ctor, -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : DiagnosticsHandler#11089138,.ctor, -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : RedirectHandler#5348816,.ctor, -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,0,0,.ctor,HttpConnectionPool http://127.0.0.1:50050 -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,0,0,GetHttp2ConnectionAsync,No available HTTP/2 connections; request queued. -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,0,0,AddHttp2ConnectionAsync,Creating new HTTP/2 connection for pool. -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,.ctor,_connectionWindow. initialCredit=65535 -> 0.023s HttpEventSourceListener - Debug: <- Event System.Net.Http - ConnectionEstablished : 2,0 -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,TraceConnection,Http2Connection(HttpConnectionPool http://127.0.0.1:50050) -> 0.023s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SetupAsync,Initial connection-level WINDOW_UPDATE, windowUpdateAmount=67043329 -> 0.024s GrpcTestContext - Information: Client writing message 0. 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=True -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReturnHttp2Connection,isNewConnection=True -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReturnHttp2Connection,Dequeued waiting HTTP/2 request. -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReturnHttp2Connection,Put HTTP/2 connection in pool. -> 0.024s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestLeftQueue : 0.9154,2,0 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendAsync,Method: POST, RequestUri: 'http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab', Version: 2.0, Content: Grpc.Net.Client.Internal.Http.PushStreamContent`2[Streaming.DataMessage,Streaming.DataMessage], Headers: { User-Agent: grpc-dotnet/2.38.0.0 TE: trailers grpc-accept-encoding: identity,gzip Transfer-Encoding: chunked traceparent: 00-28091debdcc38c27fd712dc2be0e80ae-d83d11aa4bb82e73-00 Content-Type: application/grpc } -> 0.024s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestHeadersStart : -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteHeaders, -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteIndexedHeader,index=3 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteIndexedHeader,index=6 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=17 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteIndexedHeader,index=4, value=/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteHeaderCollection, -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=2 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeaderValues,values=grpc-dotnet/2.38.0.0 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=4 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeaderValue,value=trailers -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeader,name=grpc-accept-encoding, values=identity,gzip -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeader,name=traceparent, values=00-28091debdcc38c27fd712dc2be0e80ae-d83d11aa4bb82e73-00 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteHeaderCollection, -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=2 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeaderValues,values=application/grpc -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,.ctor,[FlowControl] InitialClientStreamWindowSize: 65535, StreamWindowThreshold: 8191, WindowScaleThresholdMultiplier: 1 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Associate : HttpResponseMessage#8832854,RequestMessage,HttpResponseMessage#8832854,HttpRequestMessage#1950778 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : HttpConnectionResponseContent#12803420,.ctor, -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Associate : HttpResponseMessage#8832854,Content,HttpResponseMessage#8832854,HttpConnectionResponseContent#12803420 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=242 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendHeadersAsync,Started writing. Total header bytes=233 -> 0.024s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,Initialize,Method: POST, RequestUri: 'http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab', Version: 2.0, Content: Grpc.Net.Client.Internal.Http.PushStreamContent`2[Streaming.DataMessage,Streaming.DataMessage], Headers: { User-Agent: grpc-dotnet/2.38.0.0 TE: trailers grpc-accept-encoding: identity,gzip traceparent: 00-28091debdcc38c27fd712dc2be0e80ae-d83d11aa4bb82e73-00 Content-Type: application/grpc }, initialWindowSize=65535 -> 0.025s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendHeadersAsync,Wrote HEADERS frame. Length=233, flags=EndHeaders -> 0.025s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestHeadersStop : -> 0.025s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendRequestBodyAsync,Grpc.Net.Client.Internal.Http.PushStreamContent`2[Streaming.DataMessage,Streaming.DataMessage] -> 0.025s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestContentStart : -> 0.025s HttpEventSourceListener - Debug: <- Event System.Net.Http - ResponseHeadersStart : -> 0.025s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=0 -> 0.025s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=242 -> 0.025s Grpc.Net.Client.Internal.RetryCall - Trace: Sending 0 buffered messages from previous failed gRPC calls. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB63317JMTI" accepted. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Connections - Debug: Connection id "0HMB63317JMTI" started. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received SETTINGS frame for stream ID 0 with length 12 and flags NONE. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0. 0.025s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received HEADERS frame for stream ID 1 with length 233 and flags END_HEADERS. 0.025s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request starting HTTP/2 POST http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab application/grpc - 0.026s SERVER Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware - Debug: Request matched endpoint 'gRPC - /DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab' 0.026s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executing endpoint 'gRPC - /DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab' 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 0: StreamId=0; Type=Settings; Flags=None; PayloadLength=18. -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ChangeMaxConcurrentStreams,newValue=100, _streamsInUse=1, _availableStreamsWaiter?=False -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SignalAvailableStreamsWaiter,result=True, _availableStreamsWaiter?=False -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ChangeInitialWindowSize,newSize=98304 -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=False -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 1: StreamId=0; Type=WindowUpdate; Flags=None; PayloadLength=4. -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessWindowUpdateFrame,StreamId=0; Type=WindowUpdate; Flags=None; PayloadLength=4. amount=65537 -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,AdjustCredit,_connectionWindow. amount=65537, current=65535 -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=False -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 2: StreamId=0; Type=Settings; Flags=Ack; PayloadLength=0. -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,RefreshRtt,[FlowControl] Updated MinRtt: 4.5867 ms -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=9 -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendSettingsAckAsync,Started writing. -> 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=9 -> 0.028s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received SETTINGS frame for stream ID 0 with length 0 and flags ACK. 0.028s Grpc.Net.Client.Internal.GrpcCall - Trace: Serialized 'Streaming.DataMessage' to 3 byte message. 0.028s Grpc.Net.Client.Internal.RetryCall - Trace: Message with 8 bytes added to the buffer. There are 8 bytes buffered for this call. 0.028s Grpc.Net.Client.Internal.GrpcCall - Debug: Sending message. 0.028s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,TryRequestCreditNoLock,_connectionWindow. requested=8, current=131072, granted=8 -> 0.030s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.030s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendStreamDataAsync,Started writing. Length=17 -> 0.030s Grpc.Net.Client.Internal.GrpcCall - Trace: Message sent. 0.030s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=0 -> 0.030s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=17 -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.030s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received DATA frame for stream ID 1 with length 8 and flags NONE. 0.031s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.031s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.032s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.033s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Reading message. 0.033s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB63317JMTI", Request id "0HMB63317JMTI:00000001": started reading request body. 0.033s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Deserializing 3 byte message to 'Streaming.DataMessage'. 0.033s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Received message. 0.033s GrpcTestContext - Information: Server received 0. 0.033s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Reading message. 0.034s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.034s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.035s GrpcTestContext - Information: Client writing message 1. 0.035s Grpc.Net.Client.Internal.GrpcCall - Trace: Serialized 'Streaming.DataMessage' to 3 byte message. 0.035s Grpc.Net.Client.Internal.RetryCall - Trace: Message with 8 bytes added to the buffer. There are 16 bytes buffered for this call. 0.035s Grpc.Net.Client.Internal.GrpcCall - Debug: Sending message. 0.035s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,TryRequestCreditNoLock,_connectionWindow. requested=8, current=131064, granted=8 -> 0.035s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.035s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendStreamDataAsync,Started writing. Length=17 -> 0.035s Grpc.Net.Client.Internal.GrpcCall - Trace: Message sent. 0.035s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=0 -> 0.035s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=17 -> 0.035s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received DATA frame for stream ID 1 with length 8 and flags NONE. 0.035s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Deserializing 3 byte message to 'Streaming.DataMessage'. 0.035s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Trace: Received message. 0.036s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.036s GrpcTestContext - Information: Server failing at 1. Next failure at 2. 0.037s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Information: Error status code 'Unavailable' raised. Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="") at Grpc.AspNetCore.FunctionalTests.Client.RetryTests.<>c__DisplayClass0_0.<g__ClientStreamingWithReadFailures|0>d.MoveNext() in /home/runner/work/grpc-dotnet/grpc-dotnet/test/FunctionalTests/Client/RetryTests.cs:line 53 --- End of stack trace from previous location --- at Grpc.AspNetCore.FunctionalTests.Client.RetryTests.<>c__DisplayClass0_0.<g__ClientStreamingWithReadFailures|0>d.MoveNext() in /home/runner/work/grpc-dotnet/grpc-dotnet/test/FunctionalTests/Client/RetryTests.cs:line 47 --- End of stack trace from previous location --- at Grpc.Shared.Server.ClientStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Shared/Server/ClientStreamingServerMethodInvoker.cs:line 100 at Grpc.Shared.Server.ClientStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Shared/Server/ClientStreamingServerMethodInvoker.cs:line 109 at Grpc.AspNetCore.Server.Internal.CallHandlers.ClientStreamingServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/CallHandlers/ClientStreamingServerCallHandler.cs:line 52 at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/CallHandlers/ServerCallHandlerBase.cs:line 102 0.037s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executed endpoint 'gRPC - /DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab' 0.037s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending HEADERS frame for stream ID 1 with length 79 and flags END_STREAM, END_HEADERS. 0.037s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.037s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=False -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 3: StreamId=1; Type=Headers; Flags=Ack, EndHeaders; PayloadLength=79. -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessHeadersFrame,StreamId=1; Type=Headers; Flags=Ack, EndHeaders; PayloadLength=79 -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,OnStatus,Status code is 200 -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,OnHeader,Content-Type: application/grpc -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendPingAsync,Started writing. -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,OnHeader,Date: Mon, 23 Aug 2021 09:31:49 GMT -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=17 -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,OnHeader,Server: Kestrel -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,OnHeader,Content-Length: 0 -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,OnHeader,grpc-status: 14 -> 0.038s HttpEventSourceListener - Debug: <- Event System.Net.Http - ResponseHeadersStop : -> 0.038s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestStop : -> 0.038s Grpc.Net.Client.Internal.GrpcCall - Trace: Response headers received. 0.038s Grpc.Net.Client.Internal.GrpcCall - Information: Call failed with gRPC error status. Status code: 'Unavailable', Message: '(null)'. 0.038s Grpc.Net.Client.Internal.GrpcCall - Debug: Finished gRPC call. 0.038s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received PING frame for stream ID 0 with length 8 and flags NONE. 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendRequestBodyAsync,Failed to send request body: System.Threading.Tasks.TaskCanceledException: A task was canceled. at Grpc.Net.Client.Internal.Http.PushStreamContent`2.SerializeToStreamAsync(Stream stream, TransportContext context) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.Net.Client/Internal/Http/PushStreamContent.cs:line 67 at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in System.Net.Http.dll:token 0x60005cc+0x3fc -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendReset,Stream reset. Request=Failed, Response=Completed. -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,Complete,Stream complete. Request=Failed, Response=Completed. -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,RemoveStream, -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReleaseStream,_streamsInUse=1 -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SignalAvailableStreamsWaiter,result=True, _availableStreamsWaiter?=False -> 0.038s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=13 -> 0.039s GrpcTestContext - Information: Client writing message 2. 0.039s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.039s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.039s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,1,SendRstStreamAsync,Started writing. errorCode=Cancel -> 0.039s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=13 -> 0.039s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending PING frame for stream ID 0 with length 8 and flags ACK. 0.039s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=False -> 0.039s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received RST_STREAM frame for stream ID 1 with length 4 and flags 0x0. 0.039s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 4: StreamId=0; Type=Ping; Flags=Ack; PayloadLength=8. -> 0.039s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,RefreshRtt,[FlowControl] Updated MinRtt: 1.9024 ms -> 0.040s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request finished HTTP/2 POST http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab application/grpc - - 200 0 application/grpc 14.6551ms 0.040s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB63317JMTI", Request id "0HMB63317JMTI:00000001": done reading request body. 0.041s Grpc.Net.Client.Internal.RetryCall - Debug: Evaluated retry for failed gRPC call. Status code: 'Unavailable', Attempt: 1, Retry: True 0.041s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s Grpc.Net.Client.Internal.RetryCall - Trace: Starting retry delay of 00:00:00. 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.041s Grpc.Net.Client.Internal.RetryCall - Debug: Starting attempt 2. 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : PushStreamContent`2#58854265,.ctor, -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Associate : HttpRequestMessage#12114821,Content,HttpRequestMessage#12114821,PushStreamContent`2#58854265 -> 0.041s Grpc.Net.Client.Internal.GrpcCall - Debug: Starting gRPC call. Method type: 'ClientStreaming', URI: 'http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab'. 0.041s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestStart : http,127.0.0.1,50050,/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab,2,0,1 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,GetHttp2ConnectionAsync,Found usable HTTP/2 connection in pool. -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendAsync,Method: POST, RequestUri: 'http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab', Version: 2.0, Content: Grpc.Net.Client.Internal.Http.PushStreamContent`2[Streaming.DataMessage,Streaming.DataMessage], Headers: { User-Agent: grpc-dotnet/2.38.0.0 TE: trailers grpc-accept-encoding: identity,gzip grpc-previous-rpc-attempts: 1 Transfer-Encoding: chunked traceparent: 00-4aed036ec44c62f37c2d8f23eee7e2d4-8d1d5584a42a8d5f-00 Content-Type: application/grpc } -> 0.041s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestHeadersStart : -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteHeaders, -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteIndexedHeader,index=3 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteIndexedHeader,index=6 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=17 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteIndexedHeader,index=4, value=/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteHeaderCollection, -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=2 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeaderValues,values=grpc-dotnet/2.38.0.0 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=4 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeaderValue,value=trailers -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeader,name=grpc-accept-encoding, values=identity,gzip -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeader,name=grpc-previous-rpc-attempts, values=1 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeader,name=traceparent, values=00-4aed036ec44c62f37c2d8f23eee7e2d4-8d1d5584a42a8d5f-00 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteHeaderCollection, -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteBytes,Length=2 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,WriteLiteralHeaderValues,values=application/grpc -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,.ctor,[FlowControl] InitialClientStreamWindowSize: 65535, StreamWindowThreshold: 8191, WindowScaleThresholdMultiplier: 1 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Associate : HttpResponseMessage#41628378,RequestMessage,HttpResponseMessage#41628378,HttpRequestMessage#12114821 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Info : HttpConnectionResponseContent#60908656,.ctor, -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - Associate : HttpResponseMessage#41628378,Content,HttpResponseMessage#41628378,HttpConnectionResponseContent#60908656 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=272 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendHeadersAsync,Started writing. Total header bytes=263 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,Initialize,Method: POST, RequestUri: 'http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab', Version: 2.0, Content: Grpc.Net.Client.Internal.Http.PushStreamContent`2[Streaming.DataMessage,Streaming.DataMessage], Headers: { User-Agent: grpc-dotnet/2.38.0.0 TE: trailers grpc-accept-encoding: identity,gzip grpc-previous-rpc-attempts: 1 traceparent: 00-4aed036ec44c62f37c2d8f23eee7e2d4-8d1d5584a42a8d5f-00 Content-Type: application/grpc }, initialWindowSize=98304 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendHeadersAsync,Wrote HEADERS frame. Length=263, flags=EndHeaders -> 0.041s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestHeadersStop : -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendRequestBodyAsync,Grpc.Net.Client.Internal.Http.PushStreamContent`2[Streaming.DataMessage,Streaming.DataMessage] -> 0.041s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestContentStart : -> 0.041s HttpEventSourceListener - Debug: <- Event System.Net.Http - ResponseHeadersStart : -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=0 -> 0.041s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=272 -> 0.041s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received HEADERS frame for stream ID 3 with length 263 and flags END_HEADERS. 0.041s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request starting HTTP/2 POST http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab application/grpc - 0.042s Grpc.Net.Client.Internal.RetryCall - Trace: Sending 2 buffered messages from previous failed gRPC calls. 0.044s Grpc.Net.Client.Internal.GrpcCall - Debug: Sending message. 0.044s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,TryRequestCreditNoLock,_connectionWindow. requested=8, current=131056, granted=8 -> 0.044s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.044s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendStreamDataAsync,Started writing. Length=17 -> 0.044s Grpc.Net.Client.Internal.GrpcCall - Trace: Message sent. 0.045s Grpc.Net.Client.Internal.GrpcCall - Debug: Sending message. 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,TryRequestCreditNoLock,_connectionWindow. requested=8, current=131048, granted=8 -> 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendStreamDataAsync,Started writing. Length=17 -> 0.045s Grpc.Net.Client.Internal.GrpcCall - Trace: Message sent. 0.045s Grpc.Net.Client.Internal.GrpcCall - Trace: Serialized 'Streaming.DataMessage' to 3 byte message. 0.045s Grpc.Net.Client.Internal.RetryCall - Trace: Message with 8 bytes added to the buffer. There are 24 bytes buffered for this call. 0.045s Grpc.Net.Client.Internal.GrpcCall - Debug: Sending message. 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,TryRequestCreditNoLock,_connectionWindow. requested=8, current=131040, granted=8 -> 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendStreamDataAsync,Started writing. Length=17 -> 0.045s Grpc.Net.Client.Internal.GrpcCall - Trace: Message sent. 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=0 -> 0.045s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=51 -> 0.045s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received DATA frame for stream ID 3 with length 8 and flags NONE. 0.045s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received DATA frame for stream ID 3 with length 8 and flags NONE. 0.045s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received DATA frame for stream ID 3 with length 8 and flags NONE. 0.046s SERVER Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware - Debug: Request matched endpoint 'gRPC - /DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab' 0.046s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executing endpoint 'gRPC - /DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab' 0.046s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Debug: Reading message. 0.046s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Information: Error reading message. System.IO.IOException: The client reset the request stream. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.ReadAsync(CancellationToken cancellationToken) in Microsoft.AspNetCore.Server.Kestrel.Core.dll:token 0x6000e13+0x0 at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/PipeExtensions.cs:line 268 0.046s SERVER Grpc.AspNetCore.Server.ServerCallHandler - Error: Error when executing service method 'f641dcc8-16bf-4f33-ad7b-c61b5d971eab'. System.IO.IOException: The client reset the request stream. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.ReadAsync(CancellationToken cancellationToken) in Microsoft.AspNetCore.Server.Kestrel.Core.dll:token 0x6000e13+0x0 at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/PipeExtensions.cs:line 268 at Grpc.AspNetCore.Server.Internal.HttpContextStreamReader`1.g__MoveNextAsync|9_0(ValueTask`1 readStreamTask) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/HttpContextStreamReader.cs:line 52 at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.Net.Common/AsyncStreamReaderExtensions.cs:line 51 at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() in Grpc.Net.Common.dll:token 0x600000d+0x0 at Grpc.AspNetCore.FunctionalTests.Client.RetryTests.<>c__DisplayClass0_0.<g__ClientStreamingWithReadFailures|0>d.MoveNext() in /home/runner/work/grpc-dotnet/grpc-dotnet/test/FunctionalTests/Client/RetryTests.cs:line 47 --- End of stack trace from previous location --- at Grpc.AspNetCore.FunctionalTests.Client.RetryTests.<>c__DisplayClass0_0.<g__ClientStreamingWithReadFailures|0>d.MoveNext() in /home/runner/work/grpc-dotnet/grpc-dotnet/test/FunctionalTests/Client/RetryTests.cs:line 47 --- End of stack trace from previous location --- at Grpc.Shared.Server.ClientStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Shared/Server/ClientStreamingServerMethodInvoker.cs:line 100 at Grpc.Shared.Server.ClientStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Shared/Server/ClientStreamingServerMethodInvoker.cs:line 109 at Grpc.AspNetCore.Server.Internal.CallHandlers.ClientStreamingServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/CallHandlers/ClientStreamingServerCallHandler.cs:line 52 at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/CallHandlers/ServerCallHandlerBase.cs:line 102 0.046s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s SERVER Microsoft.AspNetCore.Routing.EndpointMiddleware - Information: Executed endpoint 'gRPC - /DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab' 0.046s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending HEADERS frame for stream ID 3 with length 108 and flags END_STREAM, END_HEADERS. 0.046s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s SERVER Microsoft.AspNetCore.Hosting.Diagnostics - Information: Request finished HTTP/2 POST http://127.0.0.1:50050/DynamicService/f641dcc8-16bf-4f33-ad7b-c61b5d971eab application/grpc - - 200 0 application/grpc 5.0765ms 0.046s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.046s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB63317JMTI", Request id "0HMB63317JMTI:00000003": started reading request body. 0.047s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.047s SERVER Microsoft.AspNetCore.Server.Kestrel - Debug: Connection id "0HMB63317JMTI", Request id "0HMB63317JMTI:00000003": done reading request body. 0.047s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.047s HttpEventSourceListener - Debug: <- Event Grpc.AspNetCore.Server - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.047s SERVER Microsoft.AspNetCore.Server.Kestrel - Information: Connection id "0HMB63317JMTI", Request id "0HMB63317JMTI:00000003": the application completed without reading the entire request body. 0.047s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending RST_STREAM frame for stream ID 3 with length 4 and flags 0x0. 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=False -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 5: StreamId=3; Type=Headers; Flags=Ack, EndHeaders; PayloadLength=108. -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessHeadersFrame,StreamId=3; Type=Headers; Flags=Ack, EndHeaders; PayloadLength=108 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnStatus,Status code is 200 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=17 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnHeader,Content-Type: application/grpc -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SendPingAsync,Started writing. -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnHeader,Date: Mon, 23 Aug 2021 09:31:49 GMT -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=17 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnHeader,Server: Kestrel -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnHeader,Content-Length: 0 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnHeader,grpc-status: 2 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnHeader,grpc-message: Exception was thrown by handler. IOException: The client reset the request stream. -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReadFrameAsync,initialFrame=False -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessIncomingFramesAsync,Frame 6: StreamId=3; Type=RstStream; Flags=None; PayloadLength=4. -> 0.047s HttpEventSourceListener - Debug: <- Event System.Net.Http - ResponseHeadersStop : -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,ProcessRstStreamFrame,protocolError=NoError -> 0.047s HttpEventSourceListener - Debug: <- Event System.Net.Http - RequestStop : -> 0.047s Grpc.Net.Client.Internal.GrpcCall - Trace: Response headers received. 0.047s Grpc.Net.Client.Internal.GrpcCall - Information: Call failed with gRPC error status. Status code: 'Unknown', Message: 'Exception was thrown by handler. IOException: The client reset the request stream.'. 0.047s Grpc.Net.Client.Internal.GrpcCall - Debug: Finished gRPC call. 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendRequestBodyAsync,Failed to send request body: System.Threading.Tasks.TaskCanceledException: A task was canceled. at Grpc.Net.Client.Internal.Http.PushStreamContent`2.SerializeToStreamAsync(Stream stream, TransportContext context) in /home/runner/work/grpc-dotnet/grpc-dotnet/src/Grpc.Net.Client/Internal/Http/PushStreamContent.cs:line 67 at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in System.Net.Http.dll:token 0x60005cc+0x3fc -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendReset,Stream reset. Request=Failed, Response=Completed. -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,Complete,Stream complete. Request=Failed, Response=Completed. -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,RemoveStream, -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ReleaseStream,_streamsInUse=1 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,SignalAvailableStreamsWaiter,result=True, _availableStreamsWaiter?=False -> 0.047s Grpc.Net.Client.Internal.RetryCall - Debug: Evaluated retry for failed gRPC call. Status code: 'Unknown', Attempt: 2, Retry: False 0.047s Grpc.Net.Client.Internal.RetryCall - Debug: Call commited. Reason: FatalStatusCode 0.047s Grpc.Net.Client.Internal.RetryCall - Trace: Stopping retry worker. 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,ProcessOutgoingFramesAsync,writeBytes=13 -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,SendRstStreamAsync,Started writing. errorCode=Cancel -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,0,FlushOutgoingBytesAsync,ActiveLength=13 -> 0.047s GrpcTestContext - Information: Client writing message 3. 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,OnReset,resetException=System.Net.Http.Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'NO_ERROR' (0x0)., resetStreamErrorCode=NoError -> 0.047s HttpEventSourceListener - Debug: <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 57842328,28398978,3,Cancel, -> 0.047s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received PING frame for stream ID 0 with length 8 and flags NONE. 0.047s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" sending PING frame for stream ID 0 with length 8 and flags ACK. 0.047s SERVER Microsoft.AspNetCore.Server.Kestrel.Http2 - Trace: Connection id "0HMB63317JMTI" received RST_STREAM frame for stream ID 3 with length 4 and flags 0x0. 0.048s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.048s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.048s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.048s HttpEventSourceListener - Debug: <- Event Grpc.Net.Client - EventCounters : System.Diagnostics.Tracing.EventPayload -> 0.048s HttpEventSourceListener - Debug: Stopping HttpEventSourceListener. 0.051s GrpcTestContext - Information: Finishing RetryTests.ClientStreaming_MultipleWritesAndRetries_Failure ```

It's long because the test is retrying multiple requests. Probably only the last request is relevant.

The CI is using a slightly older SDK: 6.0.100-rc.1.21380.19

antonfirsov commented 3 years ago

It looks like the error in https://github.com/dotnet/runtime/issues/57617#issuecomment-903193621 is different from the one in the opening post, this time HttpClient complains about the SETTINGS frame.

@JamesNK is there a client stack trace for https://github.com/dotnet/runtime/issues/57617#issuecomment-903620983? I don't see any indication of PROTOCOL_ERROR in that log. Is HttpClient replying with RST_STREAM to an RST_STREAM there?

I think we are dealing with 3 different issues here. /cc @geoffkizer

geoffkizer commented 3 years ago

It's very strange that we'd get an error related to processing the SETTINGS frame... if this was common it would blow up all sorts of things.

I wonder if we have some sort of buffer corruption issue, either on the client or the server... this would explain the PING issue as well. More detailed logs would help track this down.

JamesNK commented 3 years ago

@JamesNK is there a client stack trace for #57617 (comment)? I don't see any indication of PROTOCOL_ERROR in that log. Is HttpClient replying with RST_STREAM to an RST_STREAM there?

~It's at the top of the log:~ Never mind, different post.

Yes, I think the 3rd stack trace is different.

karelz commented 3 years ago

Logging added in 7.0 in PR #57747 and in 6.0 RC1 in PR #57998

antonfirsov commented 3 years ago

@JamesNK for clarity: you never managed to reproduce these failures locally, only on CI, right? Are the failures bound to particular OS or machine?

karelz commented 3 years ago

@JamesNK the key question is: Now that we have logging in, how can we get logs from a case when it happens to look at the logs? Any way to reproduce it on our side, or on your side? Thanks!

JamesNK commented 3 years ago

@JamesNK for clarity: you never managed to reproduce these failures locally, only on CI, right? Are the failures bound to particular OS or machine?

The opposite: I have seen it twice, both times on my local machine. Windows 11 insider perview. I tried putting the test logic in a loop and I couldn't recreate it.

@JamesNK the key question is: Now that we have logging in, how can we get logs from a case when it happens to look at the logs? Any way to reproduce it on our side, or on your side? Thanks!

Unit tests - both locally and on CI - gather logs from Kestrel, gRPC, and HttpClient. I will add them here if I see another failure.

antonfirsov commented 3 years ago

I tried putting the test logic in a loop and I couldn't recreate it.

This often indicates that issue is not independent from the rest of the test suite. What happens if you run all tests together (several times) but copy of the flaky one?

The trick I typically use:

public static readonly int[] DummyData = Enumerable.Range(0, 1000).ToArray();

[Theory]
[MemberData(nameof(DummyData))]
public void FlakyTest(int dummy) {  /* ... */ }
karelz commented 3 years ago

@JamesNK if you hit it and can gather the logs, that would be great, thanks!

karelz commented 2 years ago

@JamesNK did you happen to catch it with logs? It is not actionable for us now.

Triage: Not actionable, moving to Future for now.