bionode / bionode-watermill

💧Bionode-Watermill: A (Not Yet Streaming) Workflow Engine
https://bionode.gitbooks.io/bionode-watermill/content/
MIT License
37 stars 11 forks source link

Async vs Sync Task creation #4

Closed thejmazz closed 8 years ago

thejmazz commented 8 years ago

Failed all test cases when I switched task from being a sync function to a co.wrap(function * (){ .. }) - which returns a promise, but is nice because yielding promises is nice.

Before:

const task = Task(...)
isStream(task) // true

after:

const task = Task(...)
task.then((stream) => isStream(stream) /* true */ )

The reason it became async is for globby - I am checking the filesystem to see which files exist that match patterns in input and output. This may become negligible as tasks become able to pass values to each other. However, perhaps the performance impact of globby.sync will not be negligible when running many tasks, or tasks that have many files.

For now, I am just going to use globby.sync. As well, glob-stream may be worth a try - will still be async but perhaps glob-stream can be piped into the main Task stream.

At some point, Task may need to have a stream that wraps a stream. So that you can stream in input object into, and stream out an output info object on a .end after the .end/.push(null) of the inner stream.

thejmazz commented 8 years ago

As per de803d2, instead of wrapping the inner stream with an outer task stream (and then using a end to declare last item) I opted for just using the events API and emitting a custom event. then the join method can pick up on this and pass it on to the next task, or just call the next task

thejmazz commented 8 years ago

duplexify with placeholder (set read/write later)