Just making a note of an idea I had the other day.
Right now our graphs look something like this:
The big global barrier in the middle is a bit unfortunate, as all the workers end up blocking on the slowest intermediate write task. It would be nice if some target write tasks could get started already.
This should be fairly easy to accomplish. We could set the dependencies between intermediate and target tasks more specifically. We just have to work out the mapping between write tasks and read tasks.
Just making a note of an idea I had the other day.
Right now our graphs look something like this:
The big global barrier in the middle is a bit unfortunate, as all the workers end up blocking on the slowest intermediate write task. It would be nice if some target write tasks could get started already.
This should be fairly easy to accomplish. We could set the dependencies between intermediate and target tasks more specifically. We just have to work out the mapping between write tasks and read tasks.