Open dryajov opened 5 years ago
RLP blobs and lists encode the length in bytes at the start of the list. In general, this makes it impossible to stream a request or response because the first few bytes to transmit are not known until the end of the data.
The encoding would have to change to something a little different from RLP to enable streaming.
A slight workaround which reduces the round trip time (RTT) is to pipeline multiple requests for smaller amounts of data, which lets the sender break up its large requests, and the responder reply with those smaller data sets.
It's beneficial to pipeline anyway so that the gap between a reply and the next request is not "wasted". Pipelining a bit more adds framing and compression/encryption overhead, but breaks the flow up to reduce the observable RTT, and allows high priority messages to be inserted during those flows.
Reposting from https://github.com/ethereum/devp2p/issues/71#issuecomment-483364828.
Some benefits of streaming are:
In general, a stack that allows and encourages streaming fits really well with the overall requirements of Ethereum - i.e. sending sequences of blocks, or chunks of state are a natural fit for that. I believe this approach has great optimization potential, specially when viewed in the context of fast/warp sync.