Implicit gtest_filter fails when running 300 tests

[SaturnIC]

Checklist

• [x] The issue is about this extension and NOT about a fork.
• [x] Checked the ALL the SUPPORT document.
• [x] The latest version of the extension was used.
• [x] It is not related to remote-vscode or I checked the following issue
• [x] Imagine yourself into my position and think how hard to debug the issue without insufficient information. I understand that you have privacy concerns and I expect you to understand that this extension is developed for free. Thanks.

Describe the bug

• Running googletests on windows pc.
• All tests pass from the commandline (Just running the compiled exe)
• 9 / 300 Tests fail by running the test suite with test adapter out of vscode.
  • Found out the test invocation from test adapter uses --gtest_filter=<test-suite>.<test-case>:... parameters for every single test case(!) Running the test suite on the cli with the same filter parameters causes tests to fail on the cli as well
  • Found out the test invocation from test adapter uses --gtest_color=no parameter, disabling this parameter on the CLI lets only 1/300 fail
• Running the tests with --gtest_filter=* --gtest_color=no lets all tests pass => The gtest_filter parameter with many positive filter strings seems to cause flaky tests
• Running only the tests in the failed test suite by themselves (14 tests) in the VSCode test adapter GUI lets all those tests pass (they failed before)
• Question: Could the filter string in the "run all tests view" be disabled or replaced with * since it seems buggy to use gtest_filter with lots of tests?

[matepek]

Hello,

It can be that your tests are depending on each other and thats why it's flaky and not because of this extension. You can try command line with shuffle to reproduce the same issue. You can try running the same command line with filter the extension does and figuring out why your tests are flaky.

This filtering feature is a base of other features which would not work so I need solid proof it is not working properly to start to work on some alternatives.

Let me know if you found the issue or need more advice, meanwhile I don't consider this an issue of the extension.

[SaturnIC]

Yes the tests are dependent, but the vscode extension M U S T not change the order of running those tests, by using filters underneath the hood the cause a random execution order.

The order of executing the tests through VSCode should be the same as executing the tests without any filters.

[matepek]

Point 2

[SaturnIC]

Only beginners care about some arbitrary rules. ("Clean Code" is an egregious example for the harm caused by baseless philosophy being pushed with religious fervor)

Software is all about predictability and deterministic behavior.

And vscode gtest extension shuffling execution orders is an issue that must be fixed.

[matepek]

GoogleTest framework does not define execution order (if I'm correct), therefore developers should not depend on it.

[SaturnIC]

The order is fixed after compilation of the tests, so run them in the order they have been set in the binary.