Closed matthiasgomolka closed 3 years ago
make(parallelism = "future")
deliberately waits for the checksums of files to agree before moving on to subsequent targets (to compensate for network file system-related delays, usually NFS on clusters). Otherwise, drake
cannot be sure if downstream targets have the latest versions of upstream targets. Seems a likely explanation in your case.
Prework
Before trying to create a reprex, I wanted to make sure there is no obvious short answer. If this is not the case, I'll try to create a reprex.
Description
I noticed that very often, many of the rsessions which should be used for parallel processing in my drake plan, are idle (see screenshot).
I'm sure that the general setup with
future::plan()
beforedrake::make
is correct, because some targets really run in parallel as I expect.Is there a possibility that this is related to setting
format = "file"
for dynamic targets? Whenever I noticed that only one session is actually active,drake
was working on a dynamic target withformat = "file"
.Desired result
All dynamic subtargets should run in parallel.
Session info