MSVC Formula Modulator tests segv but only in debug build

[baconpaul]

A consistent release build of the unit tests works, but an MSVC mixed release/debug or debug build will generate a luajit crash / segv in the formula modulator tests.

[nuoun]

Backtrace from msvc for x64-debug target (ignore the extra errors, test-runner could not find resource folder)

Unhandled exception at 0x00007FF7C6311177 in surge-testrunner.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000682A009000).

call stack:
    surge-testrunner.exe!__chkstk() Line 109    Unknown
>   surge-testrunner.exe!CATCH2_INTERNAL_TEST_27() Line 466 C++
    surge-testrunner.exe!Catch::TestInvokerAsFunction::invoke() Line 6156   C++
    surge-testrunner.exe!Catch::TestCaseHandle::invoke() Line 7265  C++
    surge-testrunner.exe!Catch::RunContext::invokeActiveTestCase() Line 5500    C++
    surge-testrunner.exe!Catch::RunContext::runCurrentTest(std::string & redirectedCout, std::string & redirectedCerr) Line 5463    C++
    surge-testrunner.exe!Catch::RunContext::runTest(const Catch::TestCaseHandle & testCase) Line 5195   C++
    surge-testrunner.exe!Catch::`anonymous namespace'::TestGroup::execute() Line 1021   C++
    surge-testrunner.exe!Catch::Session::runInternal() Line 1243    C++
    surge-testrunner.exe!Catch::Session::run() Line 1174    C++
    surge-testrunner.exe!Catch::Session::run<char>(int argc, const char * const * argv) Line 5320   C++
    surge-testrunner.exe!runAllTests(int argc, char * * argv) Line 64   C++
    surge-testrunner.exe!main(int argc, char * * argv) Line 98  C++
    [External Code] 

output:
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD150.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD150.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD150.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD150.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FCDD0.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FCDD0.
The thread 672 has exited with code 0 (0x0).
The thread 28472 has exited with code 0 (0x0).
The thread 18784 has exited with code 0 (0x0).
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD080.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD080.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD080.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Tunings::TuningError at memory location 0x000000682A0F7960.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Tunings::TuningError at memory location 0x000000682A0F4FF0.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Tunings::TuningError at memory location 0x000000682A0F7960.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Tunings::TuningError at memory location 0x000000682A0F4FF0.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FCA70.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD670.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: std::filesystem::filesystem_error at memory location 0x000000682A0FC1F0.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: std::filesystem::filesystem_error at memory location 0x000000682A0F9A50.
Exception thrown at 0x00007FF82C7FB699 (KernelBase.dll) in surge-testrunner.exe: 0xE24C4A02.
Exception thrown at 0x00007FF82C7FB699 in surge-testrunner.exe: Microsoft C++ exception: Catch::TestFailureException at memory location 0x000000682A0FD8A0.
Exception thrown at 0x00007FF82C7FB699 (KernelBase.dll) in surge-testrunner.exe: 0xE24C4A03.
Exception thrown at 0x00007FF82C7FB699 (KernelBase.dll) in surge-testrunner.exe: 0xE24C4A02.
Exception thrown at 0x00007FF7C6311177 in surge-testrunner.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000682A009000).
Unhandled exception at 0x00007FF7C6311177 in surge-testrunner.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000682A009000).

[nuoun]

So I tried setting some flags in the CMake config (/O1 /Oy /Z7 /Ob1 /RTC-), moving some of the arrays in runFormula() and its struct to the heap but the only thing that fixes this so far is to set the stacksize to a whopping 16MB, which kind of feels like taking a sledgehammer to it.

Here is the CMake config for it:

# Set stack size to 16MB (0x1000000 bytes) for MSVC Debug builds
if (MSVC AND CMAKE_BUILD_TYPE STREQUAL "Debug")
  set(CMAKE_EXE_LINKER_FLAGS_DEBUG "${CMAKE_EXE_LINKER_FLAGS_DEBUG} /STACK:0x1000000")
endif()

Edit: Unfortunately the "Mono Voice Priority Modes" still breaks after this (which seems unrelated) for which I'll open a separate issue.