manage_numcalc_script stops before finishing all projects

[AudioEngineer99]

Hello,

I have the following problem:

I am running the manage_numcalc_script.py from my mesh2hrtf-tool-code folder, where all my project folders are also located. Everything runs smoothly on the first project. However, sometimes when a new project is started, I get the following error:

Started XXX (2/13, Nov XX XXXX, XX:XX:XX)
--------------------------------------------------------
Running 126/126 unfinished frequency steps in the project

Stop - not sufficient free RAM for this simulation project: Available RAM is 8.18 GB, but frequency step 126 of source 1 requires 8.97 GB.
Press Enter to exit manage_numcalc
Traceback (most recent call last):
  File "XXX\manage_numcalc_script.py", line 106, in <module>
    m2h.manage_numcalc(
  File "XXX\manage_numcalc.py", line 275, in manage_numcalc
    _raise_error((
  File "XXX\manage_numcalc.py", line 426, in _raise_error
    raise Exception("manage_numcalc was stopped due to an error")
Exception: manage_numcalc was stopped due to an error

So, I am running into this (line 275 of manage_numcalc.py)

ram_available, ram_used = _get_current_ram(ram_offset)
        if ram_available < instances_to_run[-1, 3] * ram_safety_factor:
            # note: it IS possible to run simulations that use even more than
            # 100% of available system RAM - only the performance will be poor.
            _raise_error((
                f"Stop - not sufficient free RAM for this simulation project: "
                f"Available RAM is {round(ram_available, 2)} GB, but frequency"
                f" step {int(instances_to_run[-1, 1])} of source "
                f"{int(instances_to_run[-1, 0])} requires "
                f"{round(instances_to_run[-1, 3] * ram_safety_factor, 2)} "
                "GB."), text_color_red, log_file, confirm_errors)

The problem can easily be fixed by just restarting manage_numcalc_script.py. Still, it remains a huge problem for me, since I have a lot of projects to run, which I usually do over night. And if the simulation just stops after one project, I am loosing a lot of time. I can also not supervise it 24/7 and restart it, whenever this happens.

Does anyone have an idea for a quick fix? I thought about adapting the input parameters of manage_numcalc_script.py (currently I am using the default ones). Maybe reducing the ram_safety_factor? I also thought about "dirty" solutions, like putting the manage_numcalc in an infinite loop with a "try/except" statement.

[f-brinkmann]

You can try to adapt the ram_safety_factor - this might help if you are missing only a tiny bit of RAM but won't get you far in most cases, I fear. To look for problems in advance, you can run numcalc --estimate_ram in all projects and check how much memory is required (multiplied with ram_safety_factor). This will give you a list of projects you can not run. The only real help with those cases is getting more RAM. A solution that would trade of accuracy for feasability is to apply a stronger mesh-grading, i.e., to use longer edge lengths in your mesh.

[AudioEngineer99]

Ok, thank you. I already have 128 GB of RAM, which numcalc fully utilizes, using all 24 CPU cores. So, I can run a single project rather quickly without problems. My problem is mainly that I cannot process several projects sequentially, since the simulation is interrupted by the above error, whenever it "jumps" to a new project. It seems that a new project is started and since the RAM might still be occupied by the previous project, it just stops, raising the above error.

Could it help to increase the wait_time parameter maybe?

[f-brinkmann]

Are you using the latest version from https://github.com/Any2HRTF/Mesh2HRTF/tree/master/mesh2hrtf/NumCalc? If I remember correctly, we also discovered this problem and hopefully fixed it. Did not fully understnad it in your first description...

[AudioEngineer99]

I think, I am not using the most recent version then. I will try that. Thank you.

[AudioEngineer99]

Have there been changes to the actual solver from v1.0.0 to v1.1.1?

In other words, are there differences to be expected in the final HRTFs between v1.0.0 and v1.1.1 of Mesh2HRTF? I am conducting a study and I am hesitant in changing the mesh2hrtf version on the way.

[f-brinkmann]

No, there weren't any changes in the NumCalc kernel. Blender files are now saved compressed, which might decrease the size of your projects without loosing any information:

https://github.com/Any2HRTF/Mesh2HRTF/blob/master/HISTORY.rst