Polyglot Support - Githubissues

benjchristensen commented 10 years ago

I suggest expanding this initiative beyond the JVM since most of us need our data streams and systems to interact over network boundaries with other languages.

Thus, it seems it's actually more important to define the protocol and contract and then allow each language platform to define the interfaces that meet it.

Perhaps an approach to this is breaking out into multiple sub projects such as:

/reactive-streams/specification (purely for contract definition)
/reactive-streams/reactive-streams-jvm
/reactive-streams/reactive-streams-dotnet
/reactive-streams/reactive-streams-javascript
/reactive-streams/reactive-streams-websockets
/reactive-streams/reactive-streams-tcp
/reactive-streams/reactive-streams-udp

Even if the focus in the short-term remains on the JVM interface design, we would gain a lot by including communities such as Javascript/Node.js, Erlang, .Net, banking and financial trading (who have been doing high performance messaging for decades). It would also make the model far more useful as we could then consume a reactive stream from Javascript in a browser via WebSockets to powered by Netty or Node.js receiving data from Rx/Akka/Reactor/whatever and it would "just work".

benjchristensen commented 10 years ago

I'd still like to see this. Focusing on getting the Java side of things figured out right now then will turn my attention to at least WebSockets.

tmontgomery commented 10 years ago

Same as @benjchristensen . I'd like to see this. And plan to work on WebSocket based protocol once I can clean a few things off my plate.

viktorklang commented 10 years ago

This sounds great, I just wanted to make sure that the Issue wasn't stale.

On Fri, Jul 18, 2014 at 8:07 PM, Todd L. Montgomery < notifications@github.com> wrote:

Same as @benjchristensen https://github.com/benjchristensen . I'd like to see this. And plan to work on WebSocket based protocol once I can clean a few things off my plate.

— Reply to this email directly or view it on GitHub https://github.com/reactive-streams/reactive-streams/issues/45#issuecomment-49461926 .

Cheers, √

rozza commented 9 years ago

Regarding dotnet I see in their design notes that they are looking at working on better solutions to async sequences / streams:

Async sequences: We introduced single-value asynchrony in C# 5, but do not yet 
have a satisfactory approach to asynchronous sequences or streams

Might be a good time to collaborate - see: https://github.com/dotnet/roslyn/issues/98

viktorklang commented 9 years ago

Hi @rozza, sorry for the belated reply. Thanks for the heads up!

jeffsteinmetz commented 9 years ago

Is it safe to say Reactive Streams are currently meant work on a single machine / single JVM?

My confusion is based on my understanding of Akka supporting remote actors (in a cluster). That said, is there an outline that describes the use case Reactive Streams are best suited for?

Specifically, can subscribers be distributed across a cluster? Or am I missing the point?

smaldini commented 9 years ago

Nope its not meant to work a single machine, its a contract over how a resource A passes bounded demand to B. Naturally the contract fits in Java threading but it is more than that. If you write a driver for a database or a message broker, or a microservice client, you will need to deal with some issues RS takes care of:

Latency: thats why the contract uses specific callbacks on subscribe, next, error and completion signals. They might happen eventually, not when a call to the protected resource is made (blocking).
Error isolation: the error signal carries the effective underlying errors instead of letting an implementation rising an exception.
Backpressure: the Subscriber implementation, which can be an http client driver for instance, will control how many data should be read upstream by the Publisher (or even a Processor chain, as in a data pipeline). Because it can control this demand when it has effectively sent his data over IO, it knows how many in-flight data there is. The publisher won't send anything to subscribers who didn't ask a given amount of data, naturally propagating the backpressure decision to the source.
Batching: since the demand is a long number between 1 and Long.MAX, an implementation can decide to group/aggregate events accordingly without introducing specific metadata information.

Beyond that the contract specifies additional control over when to start and stop consuming (Subscription protocol). E.g., a database driver will close is connection if a Subscriber invoked Subscription.cancel.

Now all these classic software components (the drivers etc) can implement the same contract to pass these signals around, this is where the win occurs for all of us implementors and users. Because we have a single contracts, database drivers, message broker drivers, reactive systems including reactive extensions, clients, servers, they can all be bound in a single processing chain (calling subscribe in chain) to compose a fully asynchronous pipeline with flow-control. Obviously adding IO protocol to carry these signals would be a plus to the spec as we have to implement this for every new transport, but a good chunk of the flow can end up being truly reactive right now.

Hope it clarifies a bit !

jeffsteinmetz commented 9 years ago

Thank you for the quick response.

So the "current" status of Reactive Streams does allow passing messages to remote actors on remote servers (I want to make sure we are talking about the current implementation vs the discussion in this thread to create new protocols.)

To be specific, here is a proposed scenario using the "current" version of RS:

A REST front end, which is a Play Application, receives with several JSON posts (lots of streaming docs or batches of docs) being sent to it from many remote sources. Realizing this could also be an AKKA HTTP endpoint, but in our case, we use Play everywhere upstream.

Logical Outline might be: This Play Framework initiates the source of the stream.

Then it calls an actor (which may or may not be on the same machine) which is the Publisher of the stream Then several subscribers to this stream may live anywhere on the cluster.

1 subscriber (actually 2 or more for redundancy) might simply drop the JSON in an S3 bucket on amazon,

another subscriber (again, actually 2 or more for redundancy) on more servers might do some ETL / Data cleaning, and drop some useful summary data in Cassandra, and so on, and so on..

I am scoping out some ideas on when a simple load balanced front end (a Play app that does everything, but several load balanced with some supporting app tier micro services) might be enough, vs building out a Play -> akka cluster vs building out Play -> Akka Streaming Cluster.

Getting my head around a specific use case.

viktorklang commented 9 years ago

@jeffsteinmetz Hi Jeff,

some of your questions sound like they may be more suited for the Play or Akka User MLs.

The goal behind this Issue is to provide, as @smaldini says: "[…] adding IO protocol to carry these signals would be a plus to the spec as we have to implement this for every new transport, but a good chunk of the flow can end up being truly reactive right now."

Defining the "how" to transport Reactive Streams signals across the network would allow for seamless interoperability between different platforms (think JS, Python, JVM, .NET) over different networked transports.

I hope that clarifies it a bit :)

jeffsteinmetz commented 9 years ago

Makes sense. I figured the question was a bit of a broad topic to be posting within an "issue" on github. So in summary, Akka to Akka reactive streams "do" support transport in a cluster (across a network boundary).

p.s. After doing more research (and to really keep me on my toes) I see Akka has an EventBus for Pub/Sub and a related Event Stream which adds one more "way to do something" to my list. EventBus + EventStream looks unrelated to ReactiveStream, but may solve a similar problem (without back pressure).
And then - I found Akka Persistance to ensure at least one delivery, which feels very Kafka like. Lots to try. :)

benlesh commented 9 years ago

Up until this afternoon, I was under the impression this org/repository was solely to support standardization of reactive streams on the JVM. Consequently, we created this organization this afternoon (https://github.com/Reactive-Streams-Open-Standard/reactive-streams-spec) which is JavaScript specific.

Speaking with @benjchristensen this afternoon, I found out about this polyglot effort and I was wondering if you all would like to role the JavaScript effort in under your organization.

The focus of the JavaScript effort right now is as follows:

Identify a common, open standard for Reactive Streams (EventStreams/Observables) for use by consuming JS libraries
Possibly create a set of unit tests (ala Promises/A+) to help test compliance to the standards.

In the future, it would also be great to have the JavaScript effort collaborate in reactive networking standards, (being communication over HTTP/WebSockets/SSE/etc).

tmontgomery commented 9 years ago

I would love to see a JavaScript effort here, myself. And I still intend to get to some network protocols around this effort.

benjchristensen commented 9 years ago

@blesh I definitely would like to see this happen as part of reactive-streams.org, and thanks @tmontgomery for jumping in to provide support of this.

@reactive-streams/contributors I propose the following and would like your feedback and/or +1 on moving forward:

create reactive-streams/reactive-streams-javascript repo for JavaScript spec, types and TCK
rename reactive-stream/reactive-streams to reactive-streams-java so Java does not own the root name
create reactive-stream/reactive-streams-io to start defining network protocols
recreate reactive-streams/reactive-streams as a root project for a generalized spec derived from the Java version that all polyglot specs should conform to.
create separate contributors groups for each repo (Java, JavaScript and IO for now)
update of home page for reactive-streams.org that is updated to match the Reactive Streams for Java 1.0 release and embrace the polyglot aspirations.

benlesh commented 9 years ago

I'm happy to help! This is easily the most exciting set of developments in programming right now, IMO.

reactive-streams/reactive-streams-js is preferable to me*, but other than that :+1:

* It might help with searching GitHub or tabbing through directories in terminal too... (e.g. "reactive-streams-j", TAB ...)

benjchristensen commented 9 years ago

@blesh I'm fine with reactive-streams/reactive-streams-js.

@viktorklang in particular I'd like your input on this.

rkuhn commented 9 years ago

@blesh Developing the JavaScript standard and TCK under this organization is definitely a good idea, we’d love to extend the family to more than just JVM languages. Your suggested repository name sounds good to me, it matches how things are called otherwise :-)

@benjchristensen I support your proposal of renaming and integrating repositories, maybe with the slight alteration of calling the JVM dialect reactive-streams-jvm (since it is meant to be used from all JVM languages).

Should we leave the artifact names as they are, based on the reasoning that Maven central is only used for JAR-files anyway?

Concerning the website we will have to update for 1.0 anyway (ASAP, where the ‘P’ was the problem so far), so folding the polyglot aspects into that rewrite should be easy and it adds one more reason to not procrastinate ;-)

benjchristensen commented 9 years ago

reactive-streams-jvm

I'm good with that name.

Should we leave the artifact names as they are, based on the reasoning that Maven central is only used for JAR-files anyway?

Yes, I think it's okay to leave the name.

<dependency>
    <groupId>org.reactivestreams</groupId>
    <artifactId>reactive-streams</artifactId>
    <version>1.0.0</version>
</dependency>

I'm not aware of any other type of artifacts outside the JVM community that end up on Maven Central, so I don't think we need to change the artifactId to reactive-streams-jvm.

and it adds one more reason to not procrastinate

Sounds good :-)

@reactive-streams/contributors any other comments on this before we move forward with creating the new reactive-streams-js repo?

@viktorklang or @rkuhn I don't have privileges to create new repos, so one of you will have to do it.

rkuhn commented 9 years ago

@benjchristensen I added you to the Owners, created the reactive-streams-io and reactive-streams-js repositories and renamed reactive-streams-jvm. @blesh has been added to the js-admin team so that he should be able to set everything in motion on the JavaScript side of things.

I hope I didn’t overlook anything, it is getting a little late here ;-)

benlesh commented 9 years ago

:+1: thank you, @rkuhn and @benjchristensen for helping us put this together.

benlesh commented 9 years ago

@rkuhn: Just checked, I don't see any admin features on that repo... I think maybe I'm missing something.

benlesh commented 9 years ago

@rkuhn: Nevermind, I just got the org invite. Took a while to show up in my email I guess.

benjchristensen commented 9 years ago

Thanks @rkuhn for the quick action on this!

viktorklang commented 9 years ago

Looking forward to having a look when I get back from vacation :-)

Cheers, √ On 25 Feb 2015 11:46, "Ben Christensen" notifications@github.com wrote:

Thanks @rkuhn https://github.com/rkuhn for the quick action on this!

— Reply to this email directly or view it on GitHub https://github.com/reactive-streams/reactive-streams-jvm/issues/45#issuecomment-75904915 .

purplefox commented 9 years ago

Very interesting proposal, but also very ambitious.

Defining a standard messaging protocol for asynchronously moving data with back pressure.Platform/ language agnostic. Scores of language specific clients will then spring up. It could even be embedded in routers.

Sounds very much like what AMQP is/was supposed to be. I've never been a big fan of AMQP (too complex imho), but any attempt to solve the problem in a simpler/better way gets my attention :)

smaldini commented 9 years ago

Nice, we started some work around that in reactor-net. Will look when I am back from vacation too :)

Sent from my iPhone

On 26 Feb 2015, at 11:43 am, Tim Fox notifications@github.com wrote:

Very interesting proposal, but also very ambitious.

Defining a standard messaging protocol for asynchronously moving data with back pressure.Platform/ language agnostic. Scores of language specific clients will then spring up. It could even be embedded in routers.

Sounds very much like what AMQP is/was supposed to be. I've never been a big fan of AMQP (too complex imho), but any attempt to solve the problem in a simpler/better way gets my attention :)

— Reply to this email directly or view it on GitHub.

benjchristensen commented 9 years ago

I have opened an issue at https://github.com/reactive-streams/reactive-streams-io/issues/1 to start this up. Please tell me if I'm embarrassing myself and should stop, or help me move forward with it if it makes sense. I am somewhat out of my league in this area so am really seeking help and expertise, so forgive what I'm sure is naive and elementary attempt at describing what I'm seeking.

benjchristensen commented 9 years ago

@purplefox Thanks for getting involved. I would love to have your guidance and involvement on this as it is honestly more aspirational for me than my core skill set.

@tmontgomery Todd, I rely heavily on your experience, skills and expressed interest as we've talked in the past to make this happen!

tmontgomery commented 9 years ago

@benjchristensen I'm in! Will definitely help.

viktorklang commented 9 years ago

Sorry for the late reply, I am currently on vacation with limited connectivity. Looking forward to be involved when I get back!

Cheers, √ On 24 Feb 2015 07:40, "Ben Christensen" notifications@github.com wrote:

@blesh https://github.com/blesh I'm fine with reactive-streams/reactive-streams-js.

@viktorklang https://github.com/viktorklang in particular I'd like your input on this.

— Reply to this email directly or view it on GitHub https://github.com/reactive-streams/reactive-streams/issues/45#issuecomment-75675012 .

viktorklang commented 9 years ago

@reactive-streams/contributors Closing this issue since these efforts will be continued as reactive-streams subprojects!

briantopping commented 9 years ago

This issue might be worth reopening as I think there's one more working group that would be helpful to polyglot components. As Reactive Streams offers "seamless interoperability between different platforms (think JS, Python, JVM, .NET) over different networked transports", there ought to be an intelligent discovery system for components. This WG would define the metadata used across app subprojects such that components can be found in a unified manner.

Two schemes I've seen are active registries that require being pinged with a trackback URL to a publicly accessible repo (http://bower.io/search) or passive registration such as Maven Central. Since different platforms are unlikely to ever converge on a single repository over time, it seems that an active registry is a better way to go. The ability to easily discover components could eventually segue into IDE support for graphical pipelines, so it may be good to think in that direction as well.

viktorklang commented 9 years ago

Hi @briantopping,

the topics are very interesting, but seem to supersede reactive-streams-jvm, so should be opened in a more general repo? Wdyt?

briantopping commented 9 years ago

Hi @viktorklang, it seemed to be the case to open another repo, but I could have been missing something. I'm happy to start a discussion there to gauge interest and see what could be generated in the way of requirements if so.

reactive-streams / reactive-streams-jvm

Polyglot Support #45