ercius / openNCEM

A collection of packages and tools for electron microscopy data analysis supported by the National Center for Electron Microscopy facility of the Molecular Foundry
GNU General Public License v3.0
58 stars 28 forks source link

Memory caching for DM3/DM4 files and ncem2png command #11

Closed gonzalorodrigo closed 6 years ago

gonzalorodrigo commented 6 years ago

This PR brings two new features to the library:

Memory caching for DM3/DM4 file: DM3/DM4 file parsing requires reads of random positions of the file. Such I/O pattern makes read operations slower, especially if the file is hosted in a network storage system. To ease this problem, this PR pre-loads the file content in memory and provides an API that wrap the file read operations and direct them the copy cached in memory.

Main Changes:

Results: x1000 speed up while executing on a NERSC super computing reading from a network parallel file system.

ncem2png: a new command-line command to extract 2D images (in PNG format) from SER, DM3, and DM4 files. Main features:

ercius commented 6 years ago

The memorydm branch from @gonzalorodrigo was tested locally by myself and works great. Thanks for adding this feature to openNCEM!