search

hyperium / tonic

A native gRPC client & server implementation with async/await support.
https://docs.rs/tonic
MIT License
9.76k stars 997 forks source link

Internal: Unexpected EOF decoding stream #1199

Closed danchia closed 1 year ago

danchia commented 1 year ago

Bug Report

During routine operation, our service received "Internal: Unexpected EOF decoding stream" while using tonic to stream from an external API. We believe this is coming from tonic here in decode.rs

Version

tonic 0.7.1 tonic 0.8.0

Platform

amazon-eks-node-1.23-v20221112 on x86_64. kernel: 5.4.219-126.411.amzn2

Description

It looks like the most likely reason this happen is the external API server hung up midway through the stream. In which case, this seems like a routine transport error which is usually reported by tonic as unknown: transport error rather than an internal error. An internal error makes it sound like there's a tonic bug.

Could you please check my understanding of the situation? If this is an error than can be routinely encountered, perhaps we should change the ErrorKind to something other than internal?

LucioFranco commented 1 year ago

@danchia can you provide the full error and relevant logs you have?

LucioFranco commented 1 year ago

I wonder if its related to https://github.com/hyperium/tonic/issues/1183

danchia commented 1 year ago

Unfortunately, I don't have much more details to provide. The full error message I have available to me is "internal: Unexpected EOF decoding stream." and we don't have trace logging turned on in production.

LucioFranco commented 1 year ago

Yeah, without more logs its hard to say exactly what is going on. Did you have tls enabled? What tls backend are you using if so? The error messages from the transport are pretty bad tbh so I am not sure if its a bug or its just some weird interaction with a goaway and the tcp connection closing early.