Enhance manage numcalc

[f-brinkmann]

• More simple and robust RAM management
• Improved verbosity in command line output

Here are the changes in the RAM management https://github.com/Any2HRTF/Mesh2HRTF/pull/101/commits/79d07c09236bce8aad6c768331dc7df481822c66

And this is an example of the new output

Starting manage_numcalc with the following arguments [May 17 2023, 21:04:01]
----------------------------------------------------------------------------
project_path: /home/bruel/Downloads/projects_empty
numcalc_path: NumCalc
max_ram_load: 30.00 GB (31.05 GB detected)
ram_safety_factor: 1.05
max_cpu_load: 90 %
max_instances: 8 (8 cores detected)
wait_time: 15 seconds
starting_order: alternate
confirm_errors: False

NumCalc executable: NumCalc

Obtaining RAM estimates for /home/bruel/Downloads/projects_empty/right_ear/NumCalc/source_1
Obtaining RAM estimates for /home/bruel/Downloads/projects_empty/left_ear/NumCalc/source_1

Per project summary of instances that will be run
-------------------------------------------------
Detected 2 Mesh2HRTF projects in
/home/bruel/Downloads/projects_empty

128/128 frequency steps to run in 'right_ear'
128/128 frequency steps to run in 'left_ear'

Started 'right_ear' project (1/2, May 17 2023, 21:04:10)
--------------------------------------------------------
Running 128/128 unfinished frequency steps in the project

1/128 starting instance from 'right_ear' (May 17 2023, 21:04:10)
source 1, step 128, 22050.0 Hz
estimated 7.99 GB RAM of available 26.36 GB required

2/128 starting instance from 'right_ear' (May 17 2023, 21:04:25)
source 1, step 30, 5167.97 Hz
estimated 4.53 GB RAM of available 18.87 GB required

3/128 starting instance from 'right_ear' (May 17 2023, 21:04:40)
source 1, step 127, 21877.7 Hz
estimated 7.99 GB RAM of available 14.61 GB required

4/128 starting instance from 'right_ear' (May 17 2023, 21:04:55)
source 1, step 33, 5684.77 Hz
estimated 4.53 GB RAM of available 7.08 GB required

... waiting for resources and checking every second (May 17 2023, 21:05:10)
4 NumCalc instances running at 51.20% CPU load
2.88 GB RAM available (27.12 GB used), 4.76 GB required

@chris-hld, this should be pretty much identical to #100 but does not use psutil.psutil.virtual_memory().used because the documentation mentions that this is for informal purposes only. It seems to work well on my machine. Can you check it on the cluster the next days?

[chris-hld]

Seems to work fine! I would also output ram.available once in the beginning to double check the configuration, maybe after the ram detected print.

[f-brinkmann]

@chris-hld: No including in echoing the parameters max_ram_load: 30.00 GB (31.05 GB detected, 27.68 GB available)

I will wait with the patch release of this until we figured #104

[chris-hld]

I can report that the RAM resource management now works as expected. I will highlight that at least on my system 30s between call and measuring RAM again was not enough time to fully register/load. With 60s in between this seems resolved. Last commit produced this output

-----------------------------------------------------
Running 13/256 unfinished frequency steps in the project

1/13 starting instance from 'FABIAN' (May 29 2023, 10:43:37)
source 2, step 128, 22050.0 Hz
estimated 81.55 GB RAM of available 380.00 GB required

2/13 starting instance from 'FABIAN' (May 29 2023, 10:44:37)
source 2, step 106, 18260.2 Hz
estimated 56.11 GB RAM of available 302.74 GB required

3/13 starting instance from 'FABIAN' (May 29 2023, 10:45:37)
source 1, step 128, 22050.0 Hz
estimated 81.55 GB RAM of available 249.50 GB required

4/13 starting instance from 'FABIAN' (May 29 2023, 10:46:37)
source 2, step 108, 18604.7 Hz
estimated 59.77 GB RAM of available 172.23 GB required

5/13 starting instance from 'FABIAN' (May 29 2023, 10:47:38)
source 1, step 127, 21877.7 Hz
estimated 81.55 GB RAM of available 115.53 GB required

... waiting for resources and checking every second (May 29 2023, 10:48:38)
5 NumCalc instances running at 25.70% CPU load
41.32 GB RAM available (401.78 GB used), 62.76 GB required

6/13 starting instance from 'FABIAN' (May 29 2023, 12:15:36)
source 2, step 109, 18776.9 Hz
estimated 59.77 GB RAM of available 63.78 GB required

... waiting for resources and checking every second (May 29 2023, 12:16:36)
6 NumCalc instances running at 32.70% CPU load
7.2 GB RAM available (435.89 GB used), 66.73 GB required

7/13 starting instance from 'FABIAN' (May 29 2023, 12:50:50)
source 2, step 126, 21705.5 Hz
estimated 78.95 GB RAM of available 79.71 GB required

... waiting for resources and checking every second (May 29 2023, 12:51:51)
6 NumCalc instances running at 13.90% CPU load
25.52 GB RAM available (417.58 GB used), 66.73 GB required

8/13 starting instance from 'FABIAN' (May 29 2023, 13:00:02)
source 1, step 112, 19293.8 Hz
estimated 63.55 GB RAM of available 100.36 GB required

... waiting for resources and checking every second (May 29 2023, 13:01:02)
7 NumCalc instances running at 14.10% CPU load
40.16 GB RAM available (402.93 GB used), 72.47 GB required

9/13 starting instance from 'FABIAN' (May 29 2023, 13:20:59)
source 2, step 122, 21016.4 Hz
estimated 73.07 GB RAM of available 73.85 GB required

... waiting for resources and checking every second (May 29 2023, 13:21:59)
6 NumCalc instances running at 14.20% CPU load
24.96 GB RAM available (418.14 GB used), 72.47 GB required

[SDX-LV]

Hi, I have re-installed my system and currently do not even have Python installed yet. + going on a long vacation soon, So I will not be of much help during the summer.