search

vmlaker / mpipe

Python API for writing multiprocessing pipelines
http://vmlaker.github.io/mpipe
MIT License
85 stars 24 forks source link

does mpipe support the unidirectional graph pipeline/topology? #8

Open jumpoutofworld opened 9 years ago

jumpoutofworld commented 9 years ago

I've tried a pipeline like this: stage1 --> stage2 --> stage3 stage1 --> stage3

That is, stage3 receives tasks from both stage1 and stage2. I've get some wired results. does mpipe support such a pipeline?

vmlaker commented 9 years ago

Fan-in is not supported, and will have undefined results.

stage1.link(stage2.link(stage3))
stage1.link(stage3)  # This is bad.

A problem can arise with propagating the None task to signal STOP to the pipeline. For example, stage3 may receive a STOP signal from stage1, while stage2 is still processing a task. Then, when stage2 attempts to propagate it's result to stage3, the pipeline gets stuck.

I will create an issue to implement an exception error upon attempted linkage of a fan-in.