search

European-XFEL / EXtra-data

Access saved EuXFEL data
https://extra-data.rtfd.io
BSD 3-Clause "New" or "Revised" License
7 stars 14 forks source link

Detector component may fail to load data under high memory pressure #441

Open philsmt opened 1 year ago

philsmt commented 1 year ago

The reading code in extra.components tries to work around HDF problems with respect so slicing and thus reads entire datasets into memory for subsequent slicing. This relies on virtual memory tricks allowing to allocate potentially very large amounts of memory, even overcomitting physical memory.

This has shown to fail on shared nodes under high memory pressure, potentially due to different settings. Selecting down a detector object to a single train and pulse still caused MultimodKeyData.ndarray() to allocate an entire sequence worth of memory, which failed.

philsmt commented 1 year ago

This only happens when using pulse selection, e.g. here: det['image.data'].select_trains(np.s_[0]).select_pulses(np.s_[0])