search

ercius / openNCEM

A collection of packages and tools for electron microscopy data analysis supported by the National Center for Electron Microscopy facility of the Molecular Foundry
GNU General Public License v3.0
60 stars 28 forks source link

dm data not accessible after file closed using on_memory option #25

Closed ercius closed 4 years ago

ercius commented 4 years ago

Using on_memory no longer loads the image data into memory. Python crashes when trying to access the data after the file is closed. This is undesirable, since its suggested to use with context management where possible and close the file after reading the data.

This seems to be related to the way dm.fileDM.fromfile uses np.frombuffer. I am using numpy 1.18.1 on Windows 10, and this is now crashing. Maybe the behavior changed in recent versions of numpy?

Im going to explicitly load the data using np.array() to overwrite the ['data'] entry in the output dict.

@gonzalorodrigo You improved dm reading using this method. Do you have any suggestions as to why this is happening or a better way to fix it?

ercius commented 4 years ago

getMemmap is also no longer working with on_memory option enabled. I will use the filename rather than the already open file identifier to return a memmap.

ercius commented 4 years ago

This affected both dm.fileDM.getDataset() and dm.fileDM.getSlice(). Its possible it is only a Windows issue.

I made it explicit for the data to be loaded into memory using np.array(). See https://github.com/ercius/openNCEM/commit/f6df731a415e9dad0a501f75936169c89be83047 and https://github.com/ercius/openNCEM/commit/f30f10e5f59d11013276f473b78788882fa83739.