search

scipp / essnmx

Data reduction for NMX at the European Spallation Source
https://scipp.github.io/essnmx/
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Memory usage optimization #90

Open YooSunYoung opened 2 weeks ago

YooSunYoung commented 2 weeks ago

The workflow crashes a kernel in a 16 GB VISA instance.

Number of Pixels: 4915200 (McStas file) Number of Rows: ?

16 GB of memory is expected to be enough. We need to see why it crashes or see how we can optimize it.

aaronfinke commented 2 weeks ago

Please note the crash happens on a VISA instance at ILL, not ESS. However, the performance on the ESS VISA instance is very poor (might be an unrelated issue).

SimonHeybrock commented 2 weeks ago

Please add more details:

aaronfinke commented 2 weeks ago

https://confluence.ess.eu/display/~aaronfinke/ESSNMX+on+VISA+performance+comparison

SimonHeybrock commented 2 weeks ago

One thing that should in my opinion be avoided is the "merging of panels". It just adds overhead, and implies that just for the merge step we need a minimum 6x the size of a single panel in terms of memory. If all we want is write all panels to the same file (or even dataset), that can be achieved without merging as well.