Ability to halt testing and wait for developer to come up at the very point of exception handler

[tapika]

We have huge amount of tests running in azure devops. We are using C#, C++ and C++/cli and test applications are mostly written in C#.

Some of tests are flaky, so they persists with some probability - for example 20%.

This basically means that test works 4 times out of 5.

It takes time to run tests, so it could be so that reaching of problematic point could take 1-4 minutes.

Trying to reproduce a problem would mean to run same tests 5 times which expands waiting time to 5-20 minutes.

If you attach a debugger - there is also a possibility that execution of tests will slow down just enough, so that problem will not be reproducible after this.

vstest.console.exe could have some additional command line argument to halt the execution if any of tests fail.

Moreover to this - develop should be able to stop at exception point, not in catch block, so it's possible that test host application could override AppDomain.CurrentDomain.FirstChanceException or similar to catch exception before it occurs.

---

Brief analysis of vstest.console.exe functionality versus azure devops tells me that it's possible also to create a custom command line executable and fully override vstest.console.exe behavior. This can be done by responding to /ListFullyQualifiedTests and similar kind of command line arguments from here: https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/VsTestV2/vstest.ts#L330

To be able to list tests in assembly.

And to respond to plain test execution, for example:

vstest.console mydll.dll /logger:trx.

Used by azure devops - command line: https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/VsTestV2/vstest.ts#L155

Used by azure devops - usage of trx to publish the results: https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/VsTestV2/vstest.ts#L638

Test console should be able to generate trx xml file, for which it's possible for example to reuse functionality from trx2unit tool (MIT license), https://github.com/gfoidl/trx2junit/blob/master/source/trx2junit/Internal/junit2trx/TrxTestResultXmlBuilder.cs

Self made test host executable or test runner is of course one alternative as such. Please let me know if anyone will try something like this.

---

Meanwhile - would it be possible for vstest platform to provide this functionality as built-in in vstest.console.exe ?

I think if necessary this ticket could be splitted into two - one for C# support, another for C++ support. But it's important to assume that test application can be in C#, but invokes C++ code, and vice versa - C++ code, but invoking C#. All 3 types of libraries could be used - native C++ dll, mixed mode C++/cli dll, managed C# dll.

[nohwnd]

So would this work roughly?

$> vstest.console tests.dll --wait-for-debugger-on-exception 

Running tests....
test1 Passed
test2 Passed 
Attach debugger to process testhost (19723)...

I am not sure we'd be able to do that in all code. But if you are willing to PR it we can help you get that to work.

[tapika]

This could be good starting point at least.

PR as pull request ? Currently not planning to touch vstest.console, maybe later... vstest.console is also not running tests itself, it starts test host and that in a turn is performing tests ? Probably some communication channel between vstest.console <> testhost ?

[nohwnd]

Here is a prototype how you could possibly modify your tests to attach a debugger in VS when something fails. But it requires changes to your test code and does not work for just any exception. https://github.com/nohwnd/vstest-examples/tree/main/attach-debugger-to-vs

But other than that there is not a good way to do this. VS can stop at exception because it attaches a debugger. Without debugger we cannot peek into the code. And implementing the debugger would just duplicate what you can do in VS natively.

I think the best way for you is to use the standard tools to debug flaky issues. As adding a better logging, or triggering a dump on failure.