A user on MAPS wanted very fine energy step sizes (0.1% of Ei - our typical setting is 0.25% of Ei). This resulted in rather large nxspe file sizes (~1GB per file) and caused Matlab request too much memory from the kernel. As the Linux kernel has a policy to overcommit memory and we typically run Mantid together with Horace, occasionally both Mantid and Horace will actually use (access) the memory they allocated (which the kernel overcommited). This results in a kernel panic and the DAaaS VM (workspace) rebooting.
The crashes are happening during the tmp construction step of gen_sqw. In some cases we're seeing memory allocations spiking by around 50GB. The crashes stop when running in serial mode or with fewer (e.g. 2 instead of 6) workers. As a temporary solution, we're setting it so that the kernel will try to kill Matlab first if it accesses overcommited memory rather than panic.
In the longer term it would be good to try to profile the code to see where memory could be saved in the gen_sqw process.
A user on MAPS wanted very fine energy step sizes (0.1% of Ei - our typical setting is 0.25% of Ei). This resulted in rather large
nxspe
file sizes (~1GB per file) and caused Matlab request too much memory from the kernel. As the Linux kernel has a policy to overcommit memory and we typically run Mantid together with Horace, occasionally both Mantid and Horace will actually use (access) the memory they allocated (which the kernel overcommited). This results in a kernel panic and the DAaaS VM (workspace) rebooting.The crashes are happening during the
tmp
construction step ofgen_sqw
. In some cases we're seeing memory allocations spiking by around 50GB. The crashes stop when running in serial mode or with fewer (e.g.2
instead of6
) workers. As a temporary solution, we're setting it so that the kernel will try to kill Matlab first if it accesses overcommited memory rather than panic.In the longer term it would be good to try to profile the code to see where memory could be saved in the
gen_sqw
process.