search

hyperspy / hyperspyUI

A user interface for the hyperspy package. https://hyperspy.org/hyperspyUI
GNU General Public License v3.0
25 stars 15 forks source link

Loading a signal lazily #127

Open magnunor opened 7 years ago

magnunor commented 7 years ago

Currently HyperSpy supports Lazy loading, however I don't think there is a way of specifying a file to be loaded lazily in HyperSpyUI.

This would be useful for loading very large datasets, which normally would exceed the available memory on most computers.

vidartf commented 7 years ago

In older versions of HyperSpyUI, this is possible by changing the HyperSpy preference to use lazy loading by default.

BTW: What does HyperSpy do for the navigation plot of lazy signals? The signals in the UI are always plotted, so if the plotting loads the entire signal into memory to calculate the navigator, this will not really be useful here...

vidartf commented 7 years ago

Also, I made a quick implementation locally, and trying to open a signal of 500MB seems to take forever and a lot of CPU (and memory usage of 500MB+ at some point). Any idea why?

francisco-dlp commented 7 years ago

The navigator is computed lazily i.e. the chunks are retrieved from the file, processed and deleted from memory. Still, it will obviously take time and CPU. For large signals it's usually better not to plot the navigator or to use another image as navigator if possible.

In https://github.com/hyperspy/hyperspy/pull/1544 the traitsui loading widget now has a lazy checkbox.

vidartf commented 7 years ago

I also noticed that the UI's contrast controls call max/min every time a figure is activated, which will of course be expensive for navigator plots (it uses plot.data_function() to inspect min/max values). I guess this can at least be optimized to only run if the contrast controls are visible (and when being made visible), but that is only a tweak.

So lazy load in the UI should maybe only use navigator sliders then? Or put up an (optional?) dialog box asking what to do for navigator (e.g. sliders, another already loaded signal, another file, or just the standard calculated image).

francisco-dlp commented 7 years ago

The second option (dialog box) sounds good! The first one can be very limiting in some cases.

CSSFrancis commented 1 month ago

Is this something that would be easy to add back as an option. It would be nice to include this for checking 4D datasets after aqusition on a microscope.