search

microsoft / testfx

MSTest framework and adapter
MIT License
749 stars 255 forks source link

Newly created Unit Test Projects cannot be run with the Test Execution Command Line Tool #445

Closed Zymus closed 6 years ago

Zymus commented 6 years ago

Description

Tests inside unit test projects that target .Net Framework 4.5 and do not reference Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll are not runnable with the Microsoft Test Execution Command Line Tool.

Note: I'm not sure if this is the right repo to report this in, but can't find anything better.

A complete working example can be found at https://github.com/Zymus/mstest-bug

Steps to reproduce

  1. Build the solution (default configuration is Debug)
  2. cd into the build folder of the ProjectWithNugetPackages (ie. ProjectWithNugetPackages/bin/Debug)
  3. Run mstest with the test container as the built dll (mstest.exe /testcontainer:ProjectWithNugetPackages.dll)

Expected behavior

My current expectation is that, if someone creates a Unit Test Project through Visual Studio, then that project is immediately runnable through the Microsoft Test Execution Command Line Tool. However, that doesn't seem to be the case.

  1. Build the solution (default configuration is Debug)
  2. cd into the build folder of the ProjectWithLegacyReference (ie. ProjectWithLegacyReference/bin/Debug)
  3. Run mstest with the test container as the built dll (mstest.exe /testcontainer:ProjectWithLegacyReference.dll)

Output should look something like:

Microsoft (R) Test Execution Command Line Tool Version 15.0.27019.1
Copyright (c) Microsoft Corporation. All rights reserved.

Loading ProjectWithLegacyReference.dll...
Starting execution...

Results               Top Level Tests
-------               ---------------
Passed                ProjectWithLegacyReference.UnitTest1.TestMethod1
1/1 test(s) Passed

Summary
-------
Test Run Completed.
  Passed  1
  ---------
  Total   1

Actual behavior

Output of running mstest against the ProjectWithNugetPackages looks like

Microsoft (R) Test Execution Command Line Tool Version 15.0.27019.1
Copyright (c) Microsoft Corporation. All rights reserved.

Loading ProjectWithNugetPackages.dll...
Starting execution...
No tests to execute.

Environment

OS: Windows 7 Versions of the Microsoft Test Execution Command Line Tool tested against:

jayaranigarg commented 6 years ago

@Zymus : We see you are using MSTestV2 to run your tests i.e. MSTest.TestFramework and MSTest.TestAdapter. Unfortunately, MSTest V2 tests are not supported with “mstest.exe”.

We recommend you to use "dotnet test" or vstest.console.exe to run your tests from CLI.

Zymus commented 6 years ago

@jayaranigarg Would dotnet test or vstest.console.exe be able to run both projects that reference MSTestV2 via Nuget package, and projects that do not reference MSTestV2, but instead reference the legacy apis?

Zymus commented 6 years ago

Actually, just after testing it, it seems that vstest.console is able to run both v2 and legacy.

Zymus commented 6 years ago

@jayaranigarg One strange thing I've noticed is that the vstest.console.exe installed at Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow fails.

$ "D:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" ProjectWithLegacyReference.dll
Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   TestMethod1

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 0.2686 Seconds

$ "D:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" ProjectWithNugetPackages.dll
Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Warning: No test is available in ProjectWithNugetPackages.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

Information: Additionally, you can try specifying '/UseVsixExtensions' command if the test discoverer & executor is installed on the machine as vsix extensions and your installation supports vsix extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
jayaranigarg commented 6 years ago

@Zymus : In VS2015, vstest.console was picked from the path you mentioned i.e. Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow For running tests in VS2015 with MSTestv2, you will have to provide /testadapterpath:<Path to folder where Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll exists">. Your commandline will look something like this 

$ "D:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" ProjectWithNugetPackages.dll
/testadapterpath: "C:\Users\jagarg\source\repos\ProjectWithNugetPackages\ProjectWithNugetPackages\bin\Debug\"

However, if you try to run your tests in VS2017 latest, you will observe vstest.console.exe is picked up from C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe and you do do not need to give any /testadapterpath to run MSTestV2 tests.

We would highly recommend you to move to latest VS2017.

ShreyasRmsft commented 6 years ago

@Zymus any more queries here? VS2017 should allow you to run MsTestV2 tests with minimal effort. Please move there. If this is not possible please follow the workaround specified by @jayaranigarg. Lemme know if you have any follow up questions, else will close the the issue.

Zymus commented 6 years ago

@ShreyasRmsft No, that is everything I think. Using VS2017 allows us to run both legacy tests, and those using the Nuget packages.

UsamaKhalil-KSO commented 6 years ago

unbelievable ! This Issue comes in many version of MS Test. @ShreyasRmsft you could not found out the root cause ? This is happening again in VS 2017 with Test Execution Command Line Tool Version 15.0.27729.1 which was updated 30 October 2018.

ShreyasRmsft commented 6 years ago

@usamakhalil86 please provide a sample repro project along with screenshots and logs.

vagisha-nidhi commented 6 years ago

Closing this issue as tracking it here.

UsamaKhalil-KSO commented 6 years ago

@vagisha-nidhi I see you have closed this issue, which is separate then https://github.com/Microsoft/testfx/issues/512

This user complained about using VS 2017 and creating unit test project for .NET frame 4.6. Those are broken now. Kindly check with VS 2017 team to use "vstest.console.exe" when creating .NET framework 4.6 unit tests as well.

vagisha-nidhi commented 6 years ago

Hi @usamakhalil86 How are you trying to run the tests? The IDE uses vstest.console.exe to run tests. Are you facing any issues while running tests with VS 2017? Can you please provide us logs and screenshots or a repro project so that we can take a look? You can collect logs in ide by enabling Tools->Options->Test->Logging Level->Diagnostic This might help you when collecting logs using command line.