When futures are spawned on a worker (called source worker), all but 1 (or until time < highwatermark) are sent to the broker. However, if there was a previously running job on this source worker, there would also be a pending task request from this worker at the broker queue. This means that of the spawned futures, it is highly likely (especially if #jobs = #workers) that 2 are executed on the source worker even when they could have been executed separately. for simulations that are small and numerous, this makes little difference, but for simulations that are long, this doubles the execution time.
I propose the following fix (will be implemented in pull request #67):
Send ALL futures spawned to the broker
Maintain an explicit one-one correspondence between requests sent and futures received. i.e. do not spam future requests.
When futures are spawned on a worker (called source worker), all but 1 (or until time < highwatermark) are sent to the broker. However, if there was a previously running job on this source worker, there would also be a pending task request from this worker at the broker queue. This means that of the spawned futures, it is highly likely (especially if #jobs = #workers) that 2 are executed on the source worker even when they could have been executed separately. for simulations that are small and numerous, this makes little difference, but for simulations that are long, this doubles the execution time.
I propose the following fix (will be implemented in pull request #67):