run_project_tests.py: project's default_options are (sometimes) shared between unrelated test cases

[agurtovoy]

I've ran into this accidentally and was able to get a stable reproduce on Windows with the attached directory of test cases and the following patch to run_project_tests.py:

--- a/run_project_tests.py
+++ b/run_project_tests.py
@@ -638,6 +638,7 @@ def detect_tests_to_run(only: typing.List[str]) -> typing.List[typing.Tuple[str,
         ('frameworks', 'frameworks', False),
         ('nasm', 'nasm', False),
         ('wasm', 'wasm', shutil.which('emcc') is None or backend is not Backend.ninja),
+        ('default options', 'default options', False),
     ]

     if only:

To reproduce, unpack the archive into test cases/default options dir, apply the above patch, and then run the tests with python run_project_tests.py --backend ninja --only "default options".

The very last test case (5 no_static_runtime) (incorrectly) fails to build for me with meson.build:5:4: ERROR: Problem encountered: No static runtime allowed when meson is invoked through run_project_tests.py, and builds correctly when I invoke meson directly.

What seems to happen here is that b_vscrt=mtd default project option from one of the previous test cases gets passed to the test in question. I haven't dug into it any further, just thought I'd report it since it's pretty nasty to troubleshoot through.

[scivision]

Another issue where run_tests state is unexpectedly modified: #5888 possibly totally unrelated

[agurtovoy]

While trying to get to the root of failing MSVC tests in https://github.com/mesonbuild/meson/pull/5954, I also discovered that MSBuild somehow inherits/overrides command line options when project tests are run back to back.

On the surface, the symptoms appear to be the same as this issue: the failing test passes in isolation (e.g. when moved into its own directory and run using something like python run_project_tests.py --only my) and fails when run as a part of common tests, because in the latter case it ends up being compiled with command line flags seemingly "inherited" from the previous test run (specifically, /ZI).

Upon further investigation, though, it turns out that, except for the GUIDs, all of the generated VS project files for both the failing and the passing situation are identical. Also, the test passes (as expected) with ninja backend. Also, the test passes with VS backend if the test cases that are run before it are modified to have the same default project options as the test in question. So the only variable that is different here is whether MSBuild has been previously invoked from the same parent process (run_project_tests.py) with different options.

Just thought I'd leave this here if anyone else runs into something similar at some point in the future.