Dataflow and flow-based tools and libraries

[jnv]

I am afraid this may be beyond the scope of this list, however I would like to suggest a separate category for dataflow tools and frameworks such as NoFlo and Node-RED. It is not a pipeline in a strict sense, though I see the dataflow as a superset of pipeline programming. Would you like to include this topic, or do you think this should be handled in a separate list?

[brainstorm]

@jnv, those are really cool, you have my +1 on those. I bet that @pekrau would be interested in checking them out too.

[pditommaso]

I agree.

Could you please open a pull request adding them under a new category "Flow-based tools and libraries"

Thanks, p

On Tue, Aug 5, 2014 at 12:47 PM, Roman Valls Guimerà < notifications@github.com> wrote:

@jnv https://github.com/jnv, those are really cool, you have my +1 on those. I bet that @pekrau https://github.com/pekrau would be interested in checking them out too.

— Reply to this email directly or view it on GitHub https://github.com/pditommaso/awesome-pipeline/issues/2#issuecomment-51181112 .

[samuell]

I wonder if it wouldn't be better with a separate (though interlinked) list all-together for flow-based / data flow libraries, as that is indeed a superset of pipeline tools, and quite an extensive such, to the extent that a combined list might get a bit unfocused.

The pipeline tools in this repo are special in that they are kind of hard to find, as they are often created by smaller groups / labs, aiming for a more niche audience than the general purpose data flow frameworks, and so I see a big value in a focused, separate pipeline list, like the current one.

With that said, a list for data flow / flow-based libraries sounds like a great idea indeed!

[pditommaso]

Would you like to open a pull request showing your idea?

Thanks, Paolo

On Mon, Aug 3, 2015 at 9:56 PM, Samuel Lampa notifications@github.com wrote:

I wonder if it wouldn't be better with a separate (though interlinked) list all-together for flow-based / data flow libraries, as that is indeed a superset of pipeline tools, and quite an extensive such, to the extent that a combined list might get a bit unfocused.

The pipeline tools in this repo are special in that they are kind of hard to find, as they are often created by smaller groups / labs, aiming for a more niche audience than the general purpose data flow frameworks, and so I would see a big value in a focused, separate pipeline list, like the current one.

Just my 5c.

— Reply to this email directly or view it on GitHub https://github.com/pditommaso/awesome-pipeline/issues/2#issuecomment-127388345 .

[ctsa]

I will pitch my own solution to the list: https://github.com/Illumina/pyflow

Very recently open-sourced to BSD. Some of its more interesting features:

• The task graph is created dynamically, so the outcome of one task can be used specify new tasks and dependencies at runtime.
• It is recursive -- a pyflow can be used as a task in a parent pyflow.

More details: http://illumina.github.io/pyflow/

[pditommaso]

@ctsa Nice! I've added it.

[samuell]

@pditommaso Sorry, I didn't respond to your request for a PR! I guess I left without action since what I thought was a separate repository altogether (which would help getting a better github URL), so it isn't possible to do in a pull request, AFAIK! :) But if you set up a repo, I'm happy to send PRs for the tools I know! :)

[pditommaso]

Hi Samuel, no problem. But in this case it's better that you manage it directly!

Cheers, p