Proposal: Support streaming in the Twirp spec

[jacksontj]

I think this is a great addition for RPC for Go (as well as other languages!). The main thing that I expect will limit adoption is the lack of streaming support. There are mechanisms to do streaming on http/1.x (a great example is the grpc-gateway stuff) where they effectively just use chunked encoding with blobs to do the streaming.

[omar391]

Any progress on this?

[swiftyspiffy]

We looked this issue over once again, and came to the conclusion that Spencer's detailed comment in https://github.com/twitchtv/twirp/issues/70 seems to be still relevant today.

@spenczar: I'm sorry for the long silence on this topic. Part of the reason I've gone quiet is just that I've been focused on other work, but part of it is that I have always had my doubts about the suitability of streaming in Twirp, and have been grappling with that uncertainty, trying to find a design that maintains simplicity.

Streams are a dangerous feature for users

@peter-edge has expressed my deepest concern extremely well. One of Twirp's best features today is that it is very hard to use it badly. You kind of can't screw things up too much if you hand it to an inexperienced developer as a tool kit. And while you can make mistakes in API design and message design, those mistakes stay local: they largely aren't architectural or infrastructural.

Streams are different. They imply expectations about how connection state is managed. Load balancing them is really hard, which means that a decision to use streams for an API has ramifications that ripple out to the infrastructure of a system. So, streams introduce some architectural risk when recommending Twirp.

How plausible is it that users trip on that pothole? Unfortunately, I think it is quite likely. Streaming is the sort of feature whose downsides can be hard to see at first, but it sounds at a glance. "Lower latency and the ability to push data to the client? Sign me up!" But people could easily reach for it too early in order to future-proof their APIs, "just in case we need streams later," and walk themselves into an architectural hole which is very difficult to get out of.

Streams are complex and hard to implement well

In addition, streams add significant complexity to the Twirp project. The alpha branch that implements them has to parse protobuf message binary encoding directly - it can't really lean on the proto library for fiddly details of the encoding, but relies on a few low-level utility functions. This kind of terrifies me. I'm worried about maintaining something like that, for the health of the project.

It also imposes a much, much heavier burden on implementers in other languages. One of the best things about Twirp has been how quick and simple it is to write a generator in new languages. We saw Ruby, JavaScript, Java, and Rust implementations appear in a matter of days of the project being first released! I doubt we would have seen any third party implementations if they needed to do low-level manipulation of byte streams to pull out binary-encoded protobuf tag numbers.

The complexity also translates into a clumsy API. It's difficult to do this in a general way that feels really ergonomic while covering the many ways a streaming connection can break. I am still not thrilled with the Go API we designed, and expect that other language implementations would be just as difficult to get really right.

Streams are required only rarely

This is all risk associated with implementing streams. What is the reward, on the other side?

Frankly, at Twitch we have hundreds of Twirp services, and have not once found a compelling need for streams. I can think of a small number of backend systems that do use streaming communication between a client and server, but all of them have extra requirements which make Twirp unusable: some talk to hardware appliances or stream video data, which cannot be represented in Protobuf in any reasonable way. Some have extreme performance requirements which Twirp doesn't aim to meet. None of our designs would work for those niche applications.

Meanwhile, most simple streaming RPCs can be implemented in terms of request-response RPCs, so long as they don't have extreme demands on latency or number of open connections (although HTTP/2 request multiplexing mostly resolves those issues anyway!). Pagination and polling are the obvious tools here, but even algorithms like rsync are surprisingly straightforward to implement in a request-response fashion, and probably more efficient than you think if you're using http/2, since the transport is streaming anyway.

Sometimes you really do need streams for your system, like if you are replicating a data stream. Nothing else is really going to work, there. But Twirp does not need to be all things for all use-cases. There is room for specialized solutions, but streams are a special thing, perhaps too specialized for Twirp, and they have resounding architectural impact.

Maybe we shouldn't do streams

Given the above, I worry about adding streams to Twirp. I think it would be a step in the wrong direction.

I know others have been relying upon streams already, though, with some adventurous souls even using them in production. I'd like to better understand these use cases. What am I missing in the above analysis?

I think to move forward we'd need to see what's changed to improve on the core issues he pointed out: streams are dangerous, hard to implement well, and rarely required. Any new perspectives are welcomed.

To be clear, the specific implementation in that issue itself is not the important part, rather the idea of streams being implemented. Also, Spencer hasn't been active on this project for a while, so let's use this issue tracker going forward.

[paralin]

Those arguments are... weak in my opinion. People use pubsub all the time with, for example, firebase. Sure scalability might be a bit harder to solve and inexperienced developers can get it wrong. But those aren't reasons to not add it at all.

I've implemented the streaming side here: https://github.com/aperturerobotics/starpc - with no special Protobuf hacks or anything else. Just using message framing (length prefix). It can carry multiple streams over a single connection.

It does require a two way connection like WebSockets or webrtc.

[tommie]

it can't really lean on the proto library for fiddly details of the encoding, but relies on a few low-level utility functions.

Why not? Are you referring to https://github.com/twitchtv/twirp/blob/2f0faeab61e52c5cb68bc5e5b79196b9a4819254/protoc-gen-twirp/generator.go#L1477 ?

That seems like a choice, not a necessity. You could use google.protobuf.Any or bytes in a normal message, and add a second layer of coding. Or you could use concatenation with a Protobuf header that just contains the length of the following message.

I doubt we would have seen any third party implementations if they needed to do low-level manipulation of byte streams to pull out binary-encoded protobuf tag numbers.

(Disregarding how this relates to my thoughts above.) Perhaps each feature has its own time. Perhaps now that those third-party implementations exist is the right time to take the next step?

But Twirp does not need to be all things for all use-cases.

Your point here is that Twirp is designed for some use-cases, and you are happy to not extend that. This is one aspect we (as prospective users) cannot argue against. This is your choice, and it's a fair question to be asking for any FR.

On the streaming side, we already have gRPC, gRPC-Web and Connect.

[spenczar]

It's been a long time since I looked at this, wow. And I haven't worked at Twitch for several years now, but I'll chime in anyway, since I still believe that comment to be a good summary of my personal views.

---

@paralin

Those arguments are... weak in my opinion. People use pubsub all the time with, for example, firebase. Sure scalability might be a bit harder to solve and inexperienced developers can get it wrong. But those aren't reasons to not add it at all.

I don't really understand this argument, since pubsub is independent of streaming. AMQP is probably the best-known pubsub protocol, and is based on polling requests from the client. You could implement something that looks a lot like RabbitMQ with Twirp without any changes to the current version.

I've implemented the streaming side here: https://github.com/aperturerobotics/starpc - with no special Protobuf hacks or anything else. Just using message framing (length prefix). It can carry multiple streams over a single connection.

This is cool! It's great that there are multiple options out there.

---

@tommie:

it can't really lean on the proto library for fiddly details of the encoding, but relies on a few low-level utility functions.

Why not? Are you referring to

https://github.com/twitchtv/twirp/blob/2f0faeab61e52c5cb68bc5e5b79196b9a4819254/protoc-gen-twirp/generator.go#L1477

? That seems like a choice, not a necessity. You could use google.protobuf.Any or bytes in a normal message, and add a second layer of coding. Or you could use concatenation with a Protobuf header that just contains the length of the following message.

The issue was that the stream was structured like a repeated oneof, and we want to write each message in the repeated array on the fly. At least as of that writing (over 4 years ago!), the github.com/golang/protobuf library could only write a complete repeated field out - a full array. No streaming writes of partial data. So, I had to implement that write of a single chunk in the repeated field myself.

Inventing a new encoding wrapped in opaque bytes would of course work, but it introduces a different sort of complexity - complexity in specification. Twirp gets a lot of mileage out of having a very simple spec, so that's a nontrivial cost.

---

Both of these comments are responding to the least important part of my original comment, I think. The implementation complexity is way less important than the architectural complexity. Streams are still very dangerous weapons!

[paralin]

Inventing a new encoding wrapped in opaque bytes would of course work, but it introduces a different sort of complexity - complexity in specification. Twirp gets a lot of mileage out of having a very simple spec, so that's a nontrivial cost.

Have you ever used Steam by Valve Software? It uses a Protobuf message header of a known fixed size containing the length and the message type ID of the following Protobuf message in the stream. It's not complicated and not dangerous, and quite a common construction.

Every time an architecture that looks slightly different than REST is proposed, project developers tend to scream "that's dangerous!" But some of the largest systems in the world use this structure. It's OK to diverge from rest semantics sometimes.

I understand if twirp doesn't want to go this route, but the argument that streams are dangerous is in itself unfounded.

[QuantumGhost]

Maybe we should make Twirp not generating stubs for streaming RPCs in Protobuf files, and let the users write their own streaming RPC library that co-exist with Twirp?

[gurleensethi]

What about SSE (server sent events)? I know this doesn't cover the case of bi-directional communication, but at least we can have server-to-client streaming (sort of), and it works over HTTP.

[TroyKomodo]

Why can you not make use of http.Hijacker? Why do you have to stick to http standards?

So for reference:

1. Handle the request as usual.
2. If the request is a streaming request, either bidi or uni stream. Hijack the connection to get back a raw io.ReadWriter
3. Then just use some sse or something over the protocol. Or whatever the actual impl of how binary is serialized using twirp (i dont use the library :P )