stepancheg
commented
4 years ago Neither gRPC nor protobuf designed to handle message of size 4G.
gRPC protocol message length is encoded in 4 bytes (thus cannot exceed 4G).
Although protobuf does not explicitly limit message size, the typical encoding of the message in protobuf is length-delimited, and length is signed int32 IIRC.
Quoting the document:
Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy.
That said, both rust-protobuf and grpc-rust are misbehaving: they should clearly report the error instead of panicking.
Thank you for providing a good example.
In our application we experienced recently failures. Upon investigation this seems to be related to the message size which clearly exceeds 4 GiB. Yet the produced error messages are not very helpful. I've create an MVE at https://github.com/pgab/grpc-crash/tree/bytes.
I've ported the same example to hyperium/tonic that fails similarly but at least gives a proper error message. The MVE is at https://github.com/pgab/grpc-crash/tree/tonic.
Of course our application should take care of the large message and possibly use a stream in order to transport the message.