Goals and Motivation

[benjchristensen]

Due to the successful collaborations on Reactive Streams for the JVM and community involvement in Reactive Extensions (RxJava and friends) I want to pursue greater polyglot support of the Reactive Stream semantics. This includes both language specific interfaces and over-the-network protocols. I propose that we collaborate as a community to achieve the use cases I list below, along with any others I'm missing that we derive together.

In full disclosure, personally I am building systems that need polyglot, stream-oriented network interaction, primarily between Java and JavaScript in the near future. I prefer to collaborate and design the solution openly rather than reinvent yet another competing solution. I am unsatisfied with current solutions or unaware of better ones. Teams at Netflix are creating custom one-off solutions based on ReactiveX/Reactive-Stream semantics and I'd prefer we not do this in isolation. Selfishly I want the input of people far better at this domain than I am since I am out of my league in defining network protocols and interfaces in non-Java languages. I also want to avoid NIH (not-invented-here) and solve these problems across organizations and companies since systems I'm building will most likely outlive my involvement in them and community support and involvement in core, foundational networking and messaging layers is far better than home grown solutions in the long run. I expect this to slow me down in the near term, but greatly accelerate and improve the medium and long-term accomplishments, and significantly improve the final outcome.

The timelines I'm hoping for would be functioning prototypes and protocol drafts in a few months, with release candidates in 6-9 months (Q3/Q4-2015) and a GA release in early 2016. I and the team I work with at Netflix intend on building our systems along these timelines to be proving out what we design here.

Additionally, I hope for collaboration across expertise domains to allow for debate, critiques, ideas and solutions that would not occur while staying in our individual silos.

Use Cases

The intent is to enable Reactive Stream semantics for async, stream-oriented IO supporting backpressure and cancelation.

On top of protocols such as TCP, WebSockets and possibly HTTP/2 it would allow bi-directional, multiplexed communication for these semantics:

• subscribe, request(n), cancel
• onNext, onError, onComplete

Usage patterns would include:

Scalar Request, Scalar Response

This would behave similarly to RPC/IPC calls.

For example:

• UP subscribe("hello", 1) // to eliminate round-trip, the initial request(n) could be included in the subscribe
• DOWN onNext("World!")
• DOWN onComplete

  Scalar Request, Vector Response

This would behave similarly to HTTP Server-Sent-Events.

For example:

• UP subscribe("names", 100) // to eliminate round-trip, the initial request(n) could be included in the subscribe
• DOWN onNext("Dave")
• DOWN onNext("Tom")
• DOWN onNext("Sarah")
• DOWN onComplete

Or with request(n) and unsubscribe on an infinite stream:

• UP subscribe("increment", 3) // to eliminate round-trip, the initial request(n) could be included in the subscribe
• DOWN onNext(1)
• DOWN onNext(2)
• DOWN onNext(3)
• UP request(2)
• DOWN onNext(4)
• DOWN onNext(5)
• UP unsubscribe

  Bidirectional Streams

This would behave more like raw TCP or WebSockets.

The following example is very poor, but representative of desire for messaging UP with event propagation DOWN across multiple subscriptions.

• UP subscribe("user-events-XYZ", 100)
• UP subscribe("data-updates", 100)
• UP msg("eventA", "abc") // fire-and-forget a message
• DOWN onNext("user-events-XYZ: eventA Completed")
• UP msg("/do/something", "args")
• DOWN onNext("user-events-XYZ: x-updated")
• DOWN onNext("data-event: 8756-modified")

  Possible Outcomes

Intended outcomes of this pursuit are:

1) Discover there is already a solution for this and we can shut this down and use it. 2) Decide we can't agree and we go off and build our own custom things. 3) We determine this is a useful and newish thing, collaborate and build the above.

Artifacts

Following are artifacts envisioned from this collaboration during this first phase.

Network Protocol

This is expected as purely a network protocol. Due to my ignorance I can't specify more, but I expect variations for:

• binary and text (for example, into JavaScript apps it may be valuable to support text/JSON whereas interprocess Java/C/Go/etc would benefit from binary)
• unidirectional and bidirectional transport layers (TCP vs HTTP/1 vs WebSockets vs HTTP/2 etc as tranport layers)
• how serialiation and protocol negotiation should work

Ultimately the desire is for protocols to be defined that can work on top of TCP, HTTP/1, HTTP/2, WebSockets and possibly others like UDP.

Java Interfaces and Reference Implementation

Java interfaces for exposing the various use cases using Reactive Streams interfaces would be very powerful to allow a standard interop for Reactive Stream IO.

It is not expected to have anything more than interfaces defined, but a reference implementation with unit tests to prove functionality should be included.

JavaScript Interfaces and Reference Implementation

Similar desire as for Java above.

Network TCK

Along with the network protocol I would expect a test suite to validate implementations.

Moving Forward

As a first step I'd like to determine if there is sufficient interest and that this is not insane, completely naive and wrong, or reinventing something that already exists.

If we get through that part, I'll work with you all to create more concrete Github issues to start designing and making this happen.

[danarmak]

I think the core use case is "connect any existing RS publisher and subscriber across an IP network", which needs at least these features:

1. Carry any valid RS traffic, don't change the RS semantics.
2. Multiplex multiple RS streams over a single transport (if using a non-multiplexing transport), because some usecases involve subscribing to 100s/1000s of 'slow' streams and opening e.g. 1000s of sockets is wasteful.
3. Either allow for protocol extensions of some kind, or specify more (possibly optional) features, e.g. for discovery of publishers. Otherwise most deployments will need to communicate out of band as well.
4. Transport support is a bit controversial. Personally I think supporting TCP is necessary because it's the only IP-based transport that can guarantee bidi traffic will work correctly on every possible network configuration. UDP-based solutions require stateful firewalls on every hop, and even then some firewalls are configured to block returning UDP traffic.

4b. If we support TCP, we might as well generalize to all bidi duplex streams, such as pipes.

4c. It's also possible to specify use of some other protocol that can run on top of TCP, such as ZMQ, which provides transparent reconnection and native multiplexing and framing. But then, if we support non-TCP transports that ZMQ doesn't, we either forfeit those features or have to reimplement them.

Other proposed transports/usecases have been:

5. HTTP and/or Websockets, to allow integration with existing HTTP stacks and use from browsers. Personally, I like the idea that Websockets are enough: they're a bidi duplex bytestream transport which can be used from HTTP servers and browsers, and is preceded by HTTP negotiation which can include authentication etc. 6. UDP-based transports which are more efficient or scalable than TCP in some usecases. Since RS streams can't be lossy or out-of-order and have to calculate correct demand, we'd need to implement these features on top of UDP or use a protocol that does, while still being more efficient than TCP.

[benjchristensen]

I agree. I just have a nagging suspicion that long-lived HTTP streaming is likely to induce errors due to HTTP proxies or HTTP-aware firewalls applying default timeout rules and such, if both sides don't transmit anything for a while.

Yes this can happen, but I almost certainly do need this protocol over HTTP transport such as WebSockets or HTTP/2 for external use. Internally I intend to use whatever is best (TCP, UDP, Aeron, etc)

I think it should be treated as a transport session being cut unexpectedly and handled by the application.

I agree with this. This is how we deal with SSE today where we have streams open 24/7.

Reactive Streams semantics with request(n) behavior should help solve an issue that we've struggled with using SSE which is buffer bloat in proxies. For example, Amazon ELBs do not work well for SSE because the producer can fill the ELB buffers. We have seen ELBs buffer up to 15 minutes worth of one of our streams then "blow up" as we overwhelmed its memory. We had to stop using ELBs for SSE streams of this nature. Putting backpressure into the application level semantics using RS.io should help solve this. Then the only issue should be occasional disconnects, and that's fine for our application to deal with as we must always deal with that anyways.

I like the idea that Websockets are enough

If HTTP/2 can't work but Websockets does that is sufficient for me to achieve my use cases. We need to address the HTTP connectivity though as external communication (WAN, over the internet) effectively requires this to get through firewall, NATs, etc.

[benlesh]

Why would you have different results buffering Web Sockets vs SSE? How do ELB buffers treat web sockets differently?

[benjchristensen]

I have never run WebSockets through an ELB. For one thing they don't support it. The example I was giving was that the application semantics of request(n) would compose throug proxies so buffer bloat wouldn't occur. SSE is just a firehose without application level backpressure so is vulnerable to buffering.

[pk11]

We are running WebSockets behind ELB in production (using the TCP backdoor) and while we ran into all sorts of issues, we did not notice any buffering.

As per this doc

HTTP/S:

When you use HTTP (layer 7) for both front-end and back-end connections, your load balancer parses the headers in the request and terminates the connection before re-sending the request to the back-end instance(s). This is the default configuration provided by Elastic Load Balancing.

TCP/SSL:

When you use TCP (layer 4) for both front-end and back-end connections, your load balancer forwards the request to the back-end instances without modification to the headers. After getting the request, your load balancer attempts to open a TCP connection to the back-end instance on the port specified in the health check configuration. If the load balancer fails to connect with the instance at the specified port within the configured response timeout period, the instance is considered unhealthy.

As for whether HTTP/2 or WebSockets should be used for the underlying protocol. Personally, I would recommend to try to make HTTP/2 work first. Obviously, requirements may vary but WebSockets require bigger infra investment upfront.

[joelhandwell]

How is this now? Still actively discussed in other places?

[OlegDokuka]

@joelhandwell please see http://rsocket.io/

[Denton3388]

Remove permission and certs from my device. And I would like to request all data gained from my device emailed to me. You have my email address