search

3dem / relion

Image-processing software for cryo-electron microscopy
https://relion.readthedocs.io/en/latest/
GNU General Public License v2.0
447 stars 199 forks source link

Slow Particle Extraction on High Latency File Systems #789

Open barureddy opened 3 years ago

barureddy commented 3 years ago

A big bottleneck in my relion processing pipeline on Amazon Web Services is that the writing of particles stacks into the mrcs files. For higher latency network file systems such as EFS, this takes a long time. A temporary solution I have is writing particle stacks to a local ssd scratch drive using the --part_dir option and then copy them back to the correct job directory and edit the star files. In my investigation of the writing of particles (and my limited understanding of the preprocessing.cpp), it seems each particle is written out in a serial matter, which I assume waits for some type of successful write confirmation from the file system before continuing. I believe this wait on a high latency file system is the source of the poor performance. While I'm no expert a possible solution could be to store in memory all particles of a stack until completed and write them out as a single file in the end.

biochem-fan commented 3 years ago

Cannot you run many (possibly more than the number of physical CPU cores) processes in parallel to hide the latency?

While I'm no expert a possible solution could be to store in memory all particles of a stack until completed and write them out as a single file in the end.

This suggestion is valid but we don't have resources to implement this.

barureddy commented 3 years ago

Running in parallel, especially beyond the standard 8 mpi thread speeds it up, however it is not a linear improvement. I think the I/O thrashing is partially to blame.