harripj
commented
2 years ago I would like to add that I tried this same procedure with a blockfile:
data = hs.load('BLO/' + fname +'.blo', lazy=True)
center = data.data[..., 128, 128].compute()And the whole file was not loaded to memory, only the computed array.
If I call del center the allocated memory is freed pretty quickly (within a second or so in my testing). So I wonder whether some references to the data are kept in the TVIPS reader.
Again testing the TVIPS reader, I am not sure how the amount of memory is being allocated, for example:
>>> from hyperspy import api as hs
>>> data = hs.load('TVIPS/' + fname +'.tvips', lazy=True)
>>> # select cropped region from data
>>> region = data.data[:, -400:, -400:]
>>> # get size in GB
>>> region.nbytes / 1e9
6.12576
>>> vals = region.compute()
This fills up my memory (at least 24GB for a 6GB array).
Perhaps this is related to dask compute however, as if I load the whole file as:
>>> from hyperspy import api as hs
>>> fname = 'rec_+60deg_20211004_150915_000'
>>> data = hs.load('TVIPS/' + fname +'.tvips', lazy=True)
>>>region = data.data[:, :, :]
>>>region.nbytes / 1e9
10.036445184
>>> vals = region.compute()This also fills my memory (at least 24 GB):
Whereas if I don't use the lazy loader:
>>> data = hs.load('TVIPS/' + fname +'.tvips', lazy=False)Then less memory is allocated:
din14970
From @harripj in hyperspy/hyperspy#2781:
Just a follow up on this. On my Windows machine I seem to be experiencing the same behaviour as before:
If I load one of our experimental datasets (size 9.9 GB) as
data = hs.load('dset.tvips', lazy=True)the file is not loaded to memory as expected. Withlazy=Falsethe peak memory usage is double the file size before settling back to ~file size:Using the lazy
loaderagain, if I just extract the center pixel from each frame as:The whole file is loaded to memory and persists:
Interestingly the memory presists even if I call
del center; del data.Dask version:
2022.03.0NumPy version :1.21.5Hyperspy version:1.7.0.dev0, ie. local install of this PR.Originally posted by @harripj in https://github.com/hyperspy/hyperspy/issues/2781#issuecomment-1079081907