Introduce Stream API

[vesense]

Currently, there are Producer API, Consumer API for basic IO operations on message queues. For many scenes, users like to process messages in Streams.

The Stream API is lightweight, not as same as the distributed streaming systems like Apache Storm, Flink, Spark-Streaming. And its implementation can be embeded in any java applications.

It provides the most common stream operations like filter,flatMap,map,reduce,groupBy,join,count,max,min,window, etc.

It's different from the existing StreamingConsumer, which is a low level Consumer API for processing consumer positions, etc. (IMHO, the class name of StreamingConsumer should be renamed to a more reasonable one, the current might be confused for users.)

[bsideup]

@vesense I created #20 for it. If OpenMessaging provides Reactive API then such operations can be plugged in by using RxJava, Akka Streams, Reactor or any other reactive framework

[vesense]

@bsideup Thanks for your comment.

The Stream API here is non-reactive, not as same as Reactive Streams.

The reactive-streams provides Publisher, Subscriber, Subscription, Processor APIs. I think we might consider providing reactive APIs in an individual module. These APIs can be used as a high level abstraction to process messages in the message queues.

[bsideup]

@vesense but why not Reactive? Does it make sense to provide reactive and non-reactive streams?

[vesense]

@bsideup The Stream API here, like Spark's DStream, Flink's DataStream or Kafka's KStream, is just stream abstraction including basic stream operations(filter,flatMap,map,reduce,groupBy,join,count,max,min,window, etc.). This is the different point from reactive-streams's common reactive APIs(Publisher, Subscriber, Subscription, Processor). reactive-streams provides these operations depending on 'RxJava, Akka Streams, Reactor or any other reactive framework'. And another point is that it's easy and convenient to translate sql to Stream related operations when we consider supporting sql(e.g. flink-sql, kafka-ksql).

[vongosling]

@vesense how do you think about if we want to support the Stream API base on the existing atomic APIs :-)

[vesense]

@vongosling Sorry for the delay because of my busy work. I will write a google doc for the Stream API design after completing the RocketMQ-Beam integration. Thanks for your patience.