vmlaker / mpipe

Python API for writing multiprocessing pipelines
http://vmlaker.github.io/mpipe
MIT License
86 stars 25 forks source link

any way to pipeline from generators or infinite pipelines? #17

Open FarisHijazi opened 3 years ago

FarisHijazi commented 3 years ago

I'm using mpipe for computer vision processing from camera frames, it's exactly what I need, but the put() get() api is nonideal, and I can't figure out a way to have infinite tasks.

I read the documentation but I keep finding that you need to feed the tasks before running, I need to feed tasks while running

stage1.link(stage2)
...

pipe = Pipeline(...

for number in range(10):
    pipe.put(number)

I tried running a thread in the background to feed a multiprocessing.Queue but it just stops when the queue gets full, I concluded that you can't run put() while you're running get()

def fill_queue():
    while True:
        if not q.full():
            pipe.put(0)
        else:
            time.sleep(0.01)  # Rest for 10ms, we have a full queue
thread = Thread(target=fill_queue, args=())
thread.daemon = True

while True:
    if q.qsize():
       results... = q.get()

either way, something is very wrong