Specflow+ runner fails with unhandled exception when running in azure devops pipeline

anthony0lai commented 2 years ago

SpecFlow Version

3.9.8

Which test runner are you using?

SpecFlow+ Runner

Test Runner Version Number

3.9.7

.NET Implementation

.NET 5.0

Project Format of the SpecFlow project

Classic project format using packages.config

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK

SpecFlow Section in app.config or content of specflow.json

{
  "bindingCulture": {
    "language": "en-us"
  },
  "language": {
    "feature": "en-us"
  },
  "stepAssemblies": [
    {
      "assembly": "test-common"
    }
  ]
}

Test-common contains some common test steps from an nuget package in our on own azure artefactory

Issue Description

When switching my existing .net 5.0 test project from NUnit to Specflow+ runner in Azure pipeline execution during dotnet test step

2021-11-10T03:06:48.1408600+00:00:TargetFramework: .NETCoreApp,Version=v5.0
2021-11-10T03:06:48.1411543+00:00:Tfm: netcoreapp5.0
2021-11-10T03:06:48.1411645+00:00:[AssemblyCache] Trying to resolve 'TechTalk.SpecRun.Framework.Executor' in folder 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0'
2021-11-10T03:06:48.1411790+00:00:[AssemblyCache] Trying to resolve 'TechTalk.SpecRun.Framework.Executor' to 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0\TechTalk.SpecRun.Framework.Executor.dll'
2021-11-10T03:06:48.1412344+00:00:[AssemblyCache] Trying to resolve 'TechTalk.SpecRun.Framework.Executor' to 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0\TechTalk.SpecRun.Framework.Executor.exe'
2021-11-10T03:06:48.1412763+00:00:[AssemblyCache] Could not resolve 'TechTalk.SpecRun.Framework.Executor' in folder 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0'
2021-11-10T03:06:48.1612894+00:00:Thread #0: Error while unpacking executor: System.ArgumentException: Path "TechTalk.SpecRun.Framework.dll" is not an absolute path. (Parameter 'assemblyPath')
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
   at TechTalk.SpecRun.Framework.FrameworkDependent.Remoting.AssemblyLoadContextReference.CreateObjectOfTypeFromAssembly(String assemblyName, String typeFullName)
   at TechTalk.SpecRun.Framework.FrameworkDependent.Remoting.AssemblyLoadContextReference.CreateObjectOfType[T](String assemblyName, String typeFullName)
   at TechTalk.SpecRun.Framework.Execution.SharedAppDomain.SharedAppDomainTestAssemblyExecutorHost.Initialize(Int32 threadId, ITestExecutionManager executionManager, IAssemblyReference testAssembly, ITestLogger logger, String testAssemblyFullPath, String testAssemblyConfigFilePath, TestExecutionConfiguration testExecutionConfiguration, String target)
   at TechTalk.SpecRun.Framework.TestThread.InitializeExecutor(ITestAssemblyExecutor executor, DeploymentContext deploymentContext, IAssemblyReference testAssembly)
2021-11-10T03:06:48.1631614+00:00:test thread #0 error: System.ArgumentException: Path "TechTalk.SpecRun.Framework.dll" is not an absolute path. (Parameter 'assemblyPath')
   at TechTalk.SpecRun.Framework.TestThread.UnpackTestThreadExecutorInfoResult(IResult`1 testThreadExecutorInfoResult)
   at TechTalk.SpecRun.Framework.TestThread.InitializeTestThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget)
   at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget)
   at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutorForTestItem(TestItem testItem)
   at TechTalk.SpecRun.Framework.TestThread.RunNonEmptyTestSuite(CancellationToken cancellationToken, TestThreadState testThreadState)
   at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun, CancellationToken cancellationToken)
   at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager, CancellationToken cancellationToken)
2021-11-10T03:06:48.1659213+00:00:test thread error: TechTalk.SpecRun.Framework.SpecRunException: At least one test thread aborted.

All test scenarios are finally skipped due to this exception. One thing I notice is that the target framework shows .NetCoreApp 5.0, instead of expected .Net 5.0

Steps to Reproduce

The change includes

Update specflow to 3.9.8 from 3.8.14
Add reference to SpecRun.Runner (3.9.7) and SpecRun.SpecFlow (3.9.7)
Remove reference to NUnit3TestAdapter and SpecFlow.NUnit

Created Default.srprofile as below

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
<Settings projectName="Namespace.GUITest" />
<Execution stopAfterFailures="100" testThreadCount="1" testSchedulingMode="Sequential" retryCount="1" retryFor="Failing" apartmentState="STA"/>
<!-- For collecting by a SpecRun server update and enable the following element. For using the 
  collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element.
<Server serverUrl="http://specrunserver:6365" publishResults="true" />
-->
<Environment  testThreadIsolation="SharedAppDomain" framework="Net5.0"/>
<TestAssemblyPaths>
<TestAssemblyPath>Namespace.GUITest.dll</TestAssemblyPath>
</TestAssemblyPaths>
<Report copyAlsoToBaseFolder="true"/>
<DeploymentTransformation>
<Steps>
  <!-- sample config transform to change the connection string-->
  <!--<ConfigFileTransformation configFile="App.config">
    <Transformation>
      <![CDATA[<?xml version="1.0" encoding="utf-8"?>
                        <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
            <connectionStrings>
              <add name="MyDatabase" connectionString="Data Source=.;Initial Catalog=MyDatabaseForTesting;Integrated Security=True" 
                   xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
            </connectionStrings>
                        </configuration>
                    ]]>
    </Transformation>
  </ConfigFileTransformation>-->
</Steps>
</DeploymentTransformation>
</TestProfile>

The failing dotnet test step in Azure Devops yaml is

    - task: DotNetCoreCLI@2
      displayName: "Tests Execution - ${{parameters.stageName}}"
      inputs:
        command: test
        projects: '$(build.artifactstagingdirectory)/${{parameters.projectName}}/**/${{parameters.projectName}}.dll'
        publishTestResults: true
        modifyOutputPath: true
        arguments: --filter "((TestCategory=${{parameters.testFilters}})&TestCategory!=off)"

All test scenarios are always skipped, with exception as shown in description. The whole log file is attached MottMac.AnalyticalEngine.GUITest_MottMac.AnalyticalEngine.GUITest_2021-11-10T030647.log

Running the same dotnet cli command in local (by extracting the command line displayed in azure pipeline console) is all fine, with only numerous of warning that assembly Cache file are not found (likely to be due to TFM is NetCoreApp, instead of .Net)

Link to Repro Project

No response

anthony0lai commented 2 years ago

A new finding I have is that the problem no longer exists in azure pipeline if I switch my test project into .net core 3.1

SabotageAndi commented 2 years ago

Strange issue. Do you make a clean build on the build agents? Perhaps there are some leftovers with old versions that make problems.

anthony0lai commented 2 years ago

Yes both agents and build are clean.

The agents are claimed only from Microsoft as we don't host any in Azure pipeline. So they are completely clean without any previous code on the test project

FYI on detail of components in their agent - https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md

SabotageAndi commented 2 years ago

Ok, so you are using hosted agents. That makes it even more strange.

SabotageAndi commented 2 years ago

It worked with NUnit, or? Why did you switch to the SpecFlow+ Runner?

anthony0lai commented 2 years ago

It worked fine using NUnit. We decided to switch to Specflow+ runner as NUnit + Specflow can no longer support retry on fail

SabotageAndi commented 2 years ago

Ah, the retry functionality. Did you tried the one included in the VS Test task?

anthony0lai commented 2 years ago

Hit into the same problem when I tried with VSTest task previously

SpecFlowOSS / SpecFlow