When reading/writing large numbers of files in parallel on MSI, occasionally
a read/write will return "does not exist". This is somewhat mitigated by trying
to read/write times, but that is more of a bandaid approach. A better approach,
as suggested by MSI, is to batch all reads and/or writes together by having a
threaded architecture:
For stage 1,3,4, each experiment in the batch gets its own thread (this is
already there except for stage 1).
Each thread submits file read and/or write requests into a dedicate IO thread
which then does the actual IO.
When reading/writing large numbers of files in parallel on MSI, occasionally a read/write will return "does not exist". This is somewhat mitigated by trying to read/write times, but that is more of a bandaid approach. A better approach, as suggested by MSI, is to batch all reads and/or writes together by having a threaded architecture:
For stage 1,3,4, each experiment in the batch gets its own thread (this is already there except for stage 1).
Each thread submits file read and/or write requests into a dedicate IO thread which then does the actual IO.