dotnet vstest when called in publish folder skips all tests

keenth commented 4 years ago

All tests are running fine in Visual Studio but when running the tests in the command line using dotnet vstest from the publish folder all tests are skipped. I have tried to set <Environment testThreadIsolation="SharedAppDomain" /> in Default.srprofile but still facing the same issue.

SpecFlow+ Runner Version:

[X] 3.2

SpecFlow Version:

[X] 3.1
[ ] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[X] SpecFlow+Runner
[ ] MSTest
[ ] NUnit
[ ] Xunit

Version number: 3.2.15

Project Format of the SpecFlow project

[ ] Classic project format using packages.config
[ ] Classic project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[X] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[X] VS 2019
[ ] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[X] Disabled

Are the latest Visual Studio updates installed?

[X] Yes
[ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

[ ] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[ ] .NET Core 3.0
[X] .NET Core 3.1

Test Execution Method:

[ ] Visual Studio Test Explorer
[ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
[X] Command line – PLEASE SPECIFY THE FULL COMMAND LINE C:\Program Files\dotnet>dotnet vstest "C:\Drop\SpecFlow.Tests.dll"

<SpecFlow> Section in app.config or content of specflow.json

{
  "bindingCulture":
  {
    "language": "en-us"
  },
  "language":
  {
    "feature": "en-us"
  },
  "plugins": []
}

Repro Project

Issue Description

Here is the log info, then I found that the two missing DLLs are located in the folder SpecFlowPlusRunner\netcoreapp3.1 but I don't know why they are not in the root publish folder.

 ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.VisualStudio.Validation, Version=15.3.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. Invalid pointer (0x80004003 (E_POINTER))
File name: 'Microsoft.VisualStudio.Validation, Version=15.3.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at TechTalk.SpecRun.Framework.TestThread.UnpackTestThreadExecutorInfoResult(IResult`1 testThreadExecutorInfoResult)
   at TechTalk.SpecRun.Framework.TestThread.InitializeTestThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget)
   at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget)
   at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutorForTestItem(TestItem testItem)
   at TechTalk.SpecRun.Framework.TestThread.RunNonEmptyTestSuite(CancellationToken cancellationToken, TestThreadState testThreadState)
   at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun, CancellationToken cancellationToken)
   at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager, CancellationToken cancellationToken)

Steps to Reproduce

In VS publish the tests project
In the CMD, run command "C:\Program Files\dotnet>dotnet vstest "C:\Drop\SpecFlow.Tests.dll"

SabotageAndi commented 4 years ago

The problem is not the dotnet vstest command, the problem is, the publishing before.

keenth commented 4 years ago

Thank you @SabotageAndi, do you mean that during the publishing, the two missing DLLs (Microsoft.VisualStudio.Threading.dll and Microsoft.VisualStudio.Validation.dll)should be copied from SpecFlowPlusRunner\netcoreapp3.1 folder to the root publish folder? So it is the Visual Studio issue?

SabotageAndi commented 4 years ago

I think it's a SpecFlow+ Runner issue and we don't handle the case of publishing the test project correctly. Does it work if you copy these two assemblies to your publishing folder?

keenth commented 4 years ago

Yes @SabotageAndi, If I copy the two DLLs to the publishing folder, all tests are running fine.

SabotageAndi commented 4 years ago

Cool, so you have a workaround until we fixed this issue.

keenth commented 4 years ago

We run the tests in the Azure release pipeline as well after publishing. I created a new task to copy the missing DLLs and the tests are running now. Thank you @SabotageAndi for your help.

SabotageAndi commented 3 years ago

We fixed this issue with a release in the past.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

SpecFlowOSS / SpecFlow