Test Execution Host for Rust

[ctaggart]

We need to run a bunch of functional tests in Azure DevOps for the Azure SDK for Rust. Would it be possible to use the Azure DevOps VSTest@2 task to call a custom Test Execution Host? What would that configuration look like to run the custom test host? Can the custom Test Execution Host be written in Rust?

[nohwnd]

Depends on your usecase. If you are simply after taking existing test base and allowing it to run as part of a bigger vstest based run, just adding an adapter might be enough for you, because that is much smaller extension point. It only handles a handful of things:

Discovers tests in a file. Execute tests in a file. Execute a given test in a file.

Convert incoming requests from vstest protocol. Convert outgoing data to VSTest protocol.

Your adapter would be hosted in our standard test execution host that runs on dotnet, and you would have to write this part in .NET. But however you setup the communication between your test runner and this adapter is up to you. You can do jsonrpc over standard IO for example. Or use existing test runner and parse its output.

Here is a minimal example of a test adapter, you would implement discoverer and executor, and put it in a dll named *TestAdapter.dll. In the example where I am just running methods through reflection, you would have to interface somehow with your test runner and add the responses to the sink.

https://github.com/nohwnd/Intent/tree/master/Intent.TestAdapter

[ctaggart]

Thanks @nohwnd! I am interested in other extension points. It looks like I can build and use a custom vstest.console.exe in azure-pipelines.yml. It looks like a DTAExecutionHost.exe runs vstest.console.exe and tried to communicate with it over stdio and json. Any details on that extension point? Is there a way for me to try out DTAExecutionHost.exe locally?

[nohwnd]

This is not an extension point per-se, and in theory you could replace vstest.console.exe and implement the same messages, etc. But this is not a protocol that is standardized, we publish a wrapper that you run from managed code to start an instance of vstest.console and run commands on it.

Vstest.console is an "orchestrator" of the test run. In most cases it is entry-point to the test platform, it decides what needs to be done with the provided sources, starts in process and out of process datacollectors, starts testhost, or some other test runtime provider, and it processes the results they produce.

By replacing it with your own implementation you would be throwing away all of that, and you would have to implement our "protocol" only to be able to execute some other test runner. So you would not be getting any benefit back from that, only maintenance cost. Once you are at the point of replacing vstest.console it is waaaay easier for your to just skip the vstesttask and just plug in any test runner that is native to rust, and can hopefully export results in one of the compatible formats that AzDo can consume. E.g. NUnit xml for test results, and JaCoCo for code coverage.

[nohwnd]

Talked a bit more about this, it will be easier to just produce results in one of the supported formats such as NUnit Xml or trx and upload them in the upload test results, list of the supported test result formats is here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=trx%2Cyaml and similarly for Code Covreage https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops

[ctaggart]

Thank you so much for your assistance! Looks like https://github.com/johnterickson/cargo2junit may be a solution that handles the conversion.