search

nessos / Streams

A lightweight F#/C# library for efficient functional-style pipelines on streams of data.
http://nessos.github.io/Streams/
Other
381 stars 46 forks source link

Beyond the pipeline - DAG / Flow Graph / Data Flow #6

Open sirinath opened 9 years ago

sirinath commented 9 years ago

Is it possible to add support to create flows beyond a pipeline by being able to create a flow graph like in:

http://akka.io/news/2014/09/12/akka-streams-0.7-released.html

palladin commented 9 years ago

Thank you for your suggestion! Actually, I'm thinking of possible ways to combine the power of Streams with other libraries like Rx and TPL Dataflow. The API of Akka Streams looks very interesting and a good source for inspiration.

sirinath commented 9 years ago

Also have a look at: https://github.com/Hopac/Hopac

Main thing is ease of use. If you have a DSL like Akka this would be easy to use.

Also see: http://www.nextflow.io/ (DataFlow) http://noflojs.org/ (Flow Based Programming)

And to a lesser extent: http://www.infoq.com/presentations/mantis (Rx, DAG, Functional Reactive Programming)

FBP: http://nifi.incubator.apache.org/

DAG: http://flink.incubator.apache.org/

palladin commented 9 years ago

Thanks for all the wonderful resources!

sirinath commented 9 years ago

Integration with Rx and TPL is very welcome and good. But I am not sure if they are the best performing options. Have a look at Hopac for better performance.

palladin commented 9 years ago

I love Vesa's work and Hopac is definitely on my radar!

sirinath commented 9 years ago

Great

sirinath commented 9 years ago

The docs have come out. http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html?_ga=1.257897355.1234628154.1418042628

It wold be great if you can have a similar DSL.

palladin commented 9 years ago

Thanks for the info. I really like the (~>) as the F# idiomatic operator for the underlying TPL.Dataflow LinkTo method.

sirinath commented 9 years ago

Yes. This would be good for starters.

But perhaps may be explore if there are better performing option that TPL also.

sirinath commented 9 years ago

Design it so you mix and match the underling framework.

sirinath commented 9 years ago

Wondering how things are shaping up in the data flow front?

palladin commented 9 years ago

Currently my focus is on MBrace.Streams (CloudStreams) but the data flow integration is a big concern for us. Unfortunately, the local-streams data flow support must wait for the Cloud data flow plans to take shape.

sirinath commented 9 years ago

I am more interested in Cloud data flows. If this is taking time at least the local case.