Tests are not executed on VS 2019 when outputFolder attribute added to the Settings element of Default.srprofile

[go-green]

SpecFlow Version

3.9.22

Which test runner are you using?

SpecFlow+ Runner

Test Runner Version Number

3.9.7

.NET Implementation

equal or greater .NET Framework 4.6.1

Project Format of the SpecFlow project

Classic project format using packages.config

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

<specFlow>
        <stepAssemblies>
            <stepAssembly assembly="XXX.YYY.Core" />
            <stepAssembly assembly="XXX.YYY.Steps" />
        </stepAssemblies>
</specFlow>

Issue Description

Recently, We updated our automation solution to the latest spec flow and spec runner versions. We have noted that VS test explorer skip the tests when we add outputFolder attribute to the Settings element of Default.srprofile. We have a few customized *.cshtml report templates.

Before the upgrade, we used to specify a folder as below (outputName="Reports\template_one.html") inside the Report element. After the upgrade, it does not create a folder as Reports but prefixes the folder name in front of the file name as Reportstemplate_one.html. Therefore we added the outputFolder="Reports" to the Settings element of Default.srprofile and found this issue.

However, tests are running as expected from the command line.

<Report>
        <Template name="Reports\template_one.cshtml" outputName="Reports\template_one.html" />
</Report>

Steps to Reproduce

1. Update your Default.srprofile as below with an "outputFolder" attribute
2. Attempt to run tests from VS 2019 test explorer
3. Note that tests are not executed

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
    <Settings projectName="XXXX.YYYYY.ZZZZZ.Tests" projectId="{32e7f246-80b7-3333-8d76-2f7834750c69}" outputFolder="Reports" />
    <Execution stopAfterFailures="0" testThreadCount="1" testSchedulingMode="Sequential" retryCount="0" />
    <TestAssemblyPaths>
        <TestAssemblyPath>XXXX.YYYYY.ZZZZZ.Tests.dll</TestAssemblyPath>
    </TestAssemblyPaths>
    <Report>
        <Template name="Reports\template_one.cshtml" outputName="template_one.html" />
        <Template name="Reports\template_two.cshtml" outputName="template_two.html" />
        <Template name="Reports\template_three.cshtml" outputName="template_three.txt" />
    </Report>
    <Filter>!@Manual &amp; !@Deprecated</Filter>
</TestProfile>

Link to Repro Project

No response

[SabotageAndi]

Sadly is the behavior of the outputFolder slightly different between Visual Studio test Explorer (vstest) and specrun.exe. I was not able to get the behavior the same for both execution ways, because this value is used by a lot of features. :-/

We added somewhen formatting to the outputName properties, so it can be that a simple escaping of the backslash should make it work again.

<Report>
        <Template name="Reports\\template_one.cshtml" outputName="Reports\\template_one.html" />
</Report>

The configuration on the report element is the preferred one.

[SabotageAndi]

As we stopped maintaining the SpecFlow+ Runner (https://specflow.org/using-specflow/the-retirement-of-specflow-runner/) I am closing this issue.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.