A collection of packages and tools for electron microscopy data analysis supported by the National Center for Electron Microscopy facility of the Molecular Foundry
GNU General Public License v3.0
60
stars
28
forks
source link
Memory caching for DM3/DM4 files and ncem2png command #11
Memory caching for DM3/DM4 file: DM3/DM4 file parsing requires reads of random positions of the file. Such I/O pattern makes read operations slower, especially if the file is hosted in a network storage system. To ease this problem, this PR pre-loads the file content in memory and provides an API that wrap the file read operations and direct them the copy cached in memory.
Main Changes:
fileDM class pre-loads the content of the source file in memory (if "on_memory" mode).
fileDM class new wrapping methods: from_file (np.fromfile), tell (file.tell), and seek (file.seek) methods that read from a file or the copy memory.
added unittest for io on DM files.
Results: x1000 speed up while executing on a NERSC super computing reading from a network parallel file system.
ncem2png: a new command-line command to extract 2D images (in PNG format) from SER, DM3, and DM4 files. Main features:
It can process multiple files in a single call.
For images with more than two dimensions it is possible to select the method to extract a 2D image from the data: What dimensions to fix and in which value.
This PR brings two new features to the library:
Memory caching for DM3/DM4 file: DM3/DM4 file parsing requires reads of random positions of the file. Such I/O pattern makes read operations slower, especially if the file is hosted in a network storage system. To ease this problem, this PR pre-loads the file content in memory and provides an API that wrap the file read operations and direct them the copy cached in memory.
Main Changes:
Results: x1000 speed up while executing on a NERSC super computing reading from a network parallel file system.
ncem2png: a new command-line command to extract 2D images (in PNG format) from SER, DM3, and DM4 files. Main features: