search

microsoft / vstest

Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
MIT License
880 stars 316 forks source link

VsTestConsoleWrapper does not support vstest.console.dll #1887

Closed rouke-broersma closed 5 years ago

rouke-broersma commented 5 years ago

Description

We are building a mutation testing framework for dotnet framework and dotnet core. We are trying to integrate vstest using the Microsoft.TestPlatform.TranslationLayer and Microsoft.TestPlatform.Portable nuget packages. This works on windows by providing the vstest.console.exe path from the nuget package to VsTestConsoleWrapper which provides the path to the VsTestConsoleProcessManager, however this does not seem to work for vstest.console.dll because the wrapper does not detect that the dll should be run using using dotnet with vstest.console.dll as the argument instead of trying to start a process for the dll directly. The IntegrationTestBase in the Microsoft.TestPlatform.TestUtilities project contains logic to differentiate between starting a vstest.console.exe process and starting a dotnet vstest.console.dll process. We need this logic in the TranslationLayer to be able to add vstest integration in our mutation test framework.

Steps to reproduce

  1. Create a dotnet console app using packages Microsoft.TestPlatform, Microsoft.TestPlatform.Portable and Microsoft.TestPlatform.TranslationLayer.
  2. Provide vstest.console.dll from Portable package, and a logfile location to VsTestConsoleWrapper
  3. Call StartSession
  4. Call InitializeExtensions with TestAdapters from Extensions folder in Portable package
  5. Call DiscoverTests on a test project

I also have a proof of concept of our vstest integration on this branch here: https://github.com/stryker-mutator/stryker-net/tree/183-vstest-integration/src/Stryker.Core/Stryker.Core/TestRunners/vstest

Expected behavior

vstest.console.dll starts up, discovers tests in test project and returns discovered tests to TranslationLayer. Logging is available to see what vstest.console.dll executed.

Actual behavior

vstest.console.dll tries to start up, immediately fails because OS does not know how to run a dll, no logging is available because vstest.console.dll never actually started

Diagnostic logs

No logs are available because vstest.console.dll never actually started.

Environment

OS: Windows 10 Enterprise Packages and versions:

rouke-broersma commented 5 years ago

See also integration test for VsTestConsoleWrapper which has test logic to say that the dotnet core vstest runner is not supported by VsTestConsoleWrapper: https://github.com/Microsoft/vstest/blob/master/test/Microsoft.TestPlatform.AcceptanceTests/TranslationLayerTests/DiscoverTests.cs#L21 Which actually uses the TestUtilities IntegrationTestBase to test that the dotnet core runner is not supported. The integration testbase that includes logic to start up the dotnet core vstest runner..

mayankbansal018 commented 5 years ago

@Mobrockers since you already have a POC, is it possible for you to pick this issue, we will be happy to help you.

rouke-broersma commented 5 years ago

@mayankbansal018 I can probably pick this up.