pbellec
commented
9 years ago So there was a bug. The manager could in some instances overwrite the list of jobs it sent to the worker, or even attempt to write this file concurrently with the worker reading it, which resulted in a crash. This is now solved: the manager only attempts to send new jobs to workers that have already received the previous update.
Even in a configuration with enough jobs for all workers, some workers may end up with no jobs. This may be a bug, or a quirky way job buffering happens. The test was on the fMRI preprocessing pipeline, with workers slowly becoming active, rather than all at once.