scipp / essnmx

Data reduction for NMX at the European Spallation Source
https://scipp.github.io/essnmx/
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Memory monitoring. #4

Open YooSunYoung opened 9 months ago

YooSunYoung commented 9 months ago

NMX uses relatively large amount of raw data, so we might need to have an option to monitor/log memory consumption. i.e. The original script from the owner was using tracemalloc to monitor memory.

It can probably be done in a similar way to the beamlime benchmarking...?

SimonHeybrock commented 9 months ago

I don't think we should worry about such things now. NMX experiments will run a long time with not that many events so data volume should not be a problem any more than for other instruments.

We can always process chunks in wall-time if files get too large, i.e., avoid the problem instead of adding monitoring.