msdmkats
commented
4 years ago Catching OOM is generally not feasible, e.g. in .NET when you do so you must terminate the app using Environment.FailFast method which does not allow setting exit code. It is also usually too late - by that time OS might have already killed some of the processes we need.
There's no way to do a priori tests in runners, because they don't know how much memory modules will need. Hypothetically, we could extend interfaces for modules with a method that would do such a check, but that would impose an extra and most often unnecessary burden on the author of the module.
I do, however, see, how a module over consuming memory and crashing the whole benchmarking process as a result might be problematic. As a guard measure I propose to add code to the global runner that would periodically (say, every 200 ms) would check, how much memory the current benchmark runner consumes, and kill it, if it consumes more than some configurable threshold. This way we could stop runners before they hang the system or cause it to kill some important processes.
The runners should check out of memory situations (e.g. catch OOM exceptions or perform apriori tests if it's possible) and return a special exit code if such a situation appears or will able to appear. The global runner in its turn should check this exit code and don't run tests of the bigger sizes if tests of the smaller sizes can't be performed due to OOM.