search

vmlaker / mpipe

Python API for writing multiprocessing pipelines
http://vmlaker.github.io/mpipe
MIT License
86 stars 24 forks source link

Adding ability to limit input backlog on an unordered stage #1

Closed frensjan closed 10 years ago

frensjan commented 10 years ago

Hi,

I'm using the mpipe library for some I/O intensive tasks followed by some processing and then some more I/O intensive tasks. Some of my pipelines have their bottleneck at the end (i.e. the last stage(s)), however work may be piling up at a rate which causes quite some issues (stressing out some components, using a more than desirable amount of memory, etc.).

I implemented the possibility to limit the maximum size of the input TubeQ pipe on an UnorderedStage. Could you take a look at this feature and it's implementation? I've included a simple example of the idea in test/backlog.py.

I've also added some additions essentially unrelated to the above. 1) a check to prevent stage self-linking and 2) getting all results from a stage via a generator like Stage.results(...)

Thanks, Frens Jan

vmlaker commented 10 years ago

Hi Frens,

Your backlog limit is a useful and much welcomed feature. And prevention of self-linking, along with generator-implemented retrieval of results are great additions. Thank you for your work!

Best, Vel

frensjan commented 10 years ago

Hi Vel,

Thanks for the quick response and the merge! I'll try and find some time to contribute to the docs etc if that's of any help to you.

Cheers, Frens