Open jpsamaroo opened 8 months ago
Am I correct in thinking that all the necessary items except for tests are complete?
Generally yes, I think we're pretty close to this being merge-ready. There are some remaining TODOs that I need to finish, but most are reasonably small. I could definitely use help with writing tests - just validating that we can run various kinds of pipelines and that they work across multiple workers would be really useful.
Adds a
spawn_streaming
task queue to transform tasks into continuously-executing equivalents that automatically take from inputs streams/channels and put their result to an output stream/channel. Useful for processing tons of individual elements of some large (or infinite) collection.Todo:
Stream
objectStream
finish_stream(xyz; return=abc)
to return custom value (elsenothing
)mmap
?)waitany
-style input stream, taking inputs from multiple taskstake!
from input streams concurrently, andwaitall
on them before continuingput!
into output streams concurrently, andwaitall
on them before continuing