NumCalc managing class and CLI

[marclava]

I coded an alternative to the Python scripts found in https://github.com/Any2HRTF/Mesh2HRTF/tree/release/1.0.0/mesh2hrtf/NumCalc

https://gist.github.com/marclava/65c8514e5adc46664940c83b1a43aefc

Please give it a try!

(made on Linux, should work on MacOS, and maybe on Windows)

[SDX-LV]

This is quite a re-write :)
Hi, could you please summarize:

• Does this script fully replace manage_numcalc_script.py? If yes, does it wait for user input before exiting? (waiting is needed to keep window open in Windows, if manager is started by double-clicking in Explorer)
• What improvements does this code bring (speed, stability, ...)?
• Any new features?
• Any changes needed to the (video) tutorials to use this? https://github.com/Any2HRTF/Mesh2HRTF/wiki/Basic_HRTF_NumCalc_Simulation#run-numcalc-simulation

I do not plan to run any simulations at the moment, so I would only test this later.

[marclava]

It does replace manage_numcalc_script.py; the CLI is integrated, but the class is independent and can be imported.

Running "python3 numcal_manager.py -h" shows a comprehensive help.

I believe it's a bit faster than the original, because there's no waiting by default (there's an option), also because all sources are managed in a single pool of steps (my main motivation for a rewrite, and I learned a lot more in the process).

It could also be integrated to the Mesh2HRTF python module as a project script, with a little revamping of the project metadata. I tried and it works well; https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (I can publish required changes to the Mesh2HRTF project in a personal fork).

New (or changed) features (I did not check all differences, and how different are some of the same features):

• a cute CLI and logging (with color, hopefully even on Windows)
• a state file (a compressed numpy pickle) to collect and reload stats of previous execution of the script.
• a debug mode to see more logging messages
• a skip_run option (like a dry run option), to check for validations and status of the project, without resuming it.
• a clean-all option to delete all previous results (including the state file)
• a stress-test mode, mostly independent of the normal mode, added to help in the development of the script.
• a log_to_file option (disabled by default, concatenating all console messages to a log file inside the project).

The CLI does not wait for user input before exiting, so I'll add it (see added comment below). It was not tested on Windows, but it should work.

As for the video tutorials, the changes are left as an exercise. I don't use Windows (but I can pop a Windows VM when required). One of the goal was to avoid copying a script inside a Mesh2HRTF project, but if a clickable file is required there should be a way to build one; IMHO, most potential users of Mesh2HRTF probably know how to use a terminal, but I understand the use case; an option could be added to create such a clickable file to copy inside projects. (see added comment below)

Anyway, the exercise was to code a productivity tool to help running NumCalc for hours (even days) on a "small" computer (with ~16GB of RAM), that could be interrupted and resumed in a friendly way; of course "users" will always expect perfect and brainless automation of their entire life, but for Mesh2HRTH I hope this tool is good enough. :-)

[marclava]

I won't add an option to wait for user input at exit. On Windows, to run a python script in a command window that stays open after the script is done, use this construct from a Windows shortcut "cmd /k python script.py" https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cmd

[f-brinkmann]

Hi Marc, and sorry for the late response. I'm blocked for the next three weeks due to preparing the end of the semester. Our Mesh2HRTF paper has been accepted for publication, and we will have to release v1.0.0 soon. This might not give us enough time to fully evaluate your solution, which would also require to write tests for it. But in general, I don't see a problem with having different versions of the parallelization around. I will try it in any case and the least we could do is link it from the Wiki where ever manage_numcalc is mentioned - what do you think of that?

[petibub]

Dear Marc, thanks for your contribution. As Fabian mentioned, we've finalized the package and it is just about to be released. What if we include your contribution in the next release, 1.1.0, where would could clearly advertise the additional features your contribution offers?

[marclava]

Thanks for your interest, and please take your time to evaluate and decide if you'd like to integrate the script in a future release, simply add a link in your wiki, or nothing at all. It'd be nice if you could try it on different platform and situations, hopefully to reduce the number of features. I'd have to compare my script to the original one to better describe the valuable features, apart that I like it because I wrote it :)... I can develop it a bit more; a possible feature to add is distributed parallelization, because that was one of my motivation (to learn with a real use case). I wish you success with your publication, and a good semester.

[f-brinkmann]

Hi Marc, the numcalc_manager is now linked from the Mesh2HRTF wiki start page: https://github.com/Any2HRTF/Mesh2HRTF/wiki

I prefer this over including it directly in Mesh2HRTF because it might be confusing having two very similar versions of this, it would require writing tests, and also managing possible issues or changes. I hope this is fine for you