gRPC Server Sends RST_STREAM without trailers when TCP Reassembly occurs

[DerekTBrown]

What version of gRPC are you using?

v1.59.0

What version of Go are you using (go version)?

1.22.0

What operating system (Linux, Windows, …) and version?

alpine 3.18 AMD64

Error description

I have a grpc-js client that communicates with via an istio service mesh to a service implemented using the grpc-go backend. The client observes an error:

error_message_string: 13 INTERNAL: Received RST_STREAM with code 0 (Call ended without gRPC status)

Upon inspection, I see that the gRPC server is sending RST_STREAM mid-response:

I would expect to see gRPC server send a HEADERS message containing trailers indicating the cause of the failure.

A few additional anecdotes:

• Following the same TCP stream, I see numerous other requests succeeding before and after this method call. This generally leads me to rule-out TCP-level issues.
• This error happens almost exclusively with one particular gRPC method handler, so its possible that there is something specific to the server behavior or characteristics of this request (large request sizes).
• All of the cases I have seen include TCP re-assembly, so its possible this is related.

[arjan-bal]

As the rst code is 0, it looks like the server successfully responded to the request. It seems strange that the gRPC Go server didn't write the HEADER frame with the trailer metadata. This is the code that handles writing the trailer before sending RST_FRAME with code 0 https://github.com/grpc/grpc-go/blob/cf1fb0a6e81e30f7130bfb20fd40d7929c5d3363/internal/transport/http2_server.go#L1076-L1086

I did find some issues related to the http handler transport:

• https://github.com/grpc/grpc-go/issues/553
• https://github.com/golang/go/issues/54723

@DerekTBrown are you using a http handler transport using ServeHTTP or a regular one using Server.Serve?

[DerekTBrown]

@arjan-bal Thanks for the quick rely. This is using Server.Serve.

[arjan-bal]

There is another code wherein an RST_STREAM with code 0 is sent when the server receives an RST_STREAM from the client first. This doesn't seem to be the case here.

I can't figure out a reason why gRPC Go would not send the trailer just by analysing the code.

@DerekTBrown can you please provide the gRPC Go server logs from the time this issue happens? See the docs for enabling logs. If you can provide a way to repro this issue, it would be even better.