SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.24k stars 754 forks source link

TestExecution.json only 1 scenario test, tests in parallel using SpecFlow+Runner #2524

Closed namilkimfree closed 2 years ago

namilkimfree commented 2 years ago

SpecFlow Version

3.9.7

Which test runner are you using?

SpecFlow+ Runner

Test Runner Version Number

3.9.7

.NET Implementation

.NET 5.0

Project Format of the SpecFlow project

Classic project format using <PackageReference> tags

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

{
  "bindingCulture": {
    "name": "en-us"
  },
  "language": {
    "feature": "en-us"
  },
  "stepAssemblies": [
    {
      "assembly": "CommonBind"
    }
  ]
}

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
  <Settings projectName="NetmarbleWebQATeam" />
  <VSTest testRetryResults="Unified" passRateRelative="10%"/>
    <Execution stopAfterFailures="0" testThreadCount="Max" testSchedulingMode="Sequential" apartmentState="MTA" retryFor="Failing" retryCount="3" />

    <Environment testThreadIsolation="SharedAppDomain" platform="x86"/>
  <TestAssemblyPaths>
    <TestAssemblyPath>Team.dll</TestAssemblyPath>
  </TestAssemblyPaths>

    <Targets>

        <Target name="Chrome">
            <Filter>Browser_Chrome</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="Test_Browser" value="Chrome" />
            </DeploymentTransformationSteps>
        </Target>

    </Targets>

</TestProfile>

Issue Description

TestExecution.json file only 1 scenario test, tests in parallel using SpecFlow+Runner

dotnet test --configuration "Test" --filter "(TestCategory=A3&Name~target: Chrome)"

When the 6 test parallel runs are complete and you check the folder, There is only one Test Execution.json file, and the content has only one scenario content.

{"Nodes":[],"ExecutionTime":"2021-11-18T07:05:27.4377052Z","GenerationTime":"0001-01-01T00:00:00","PluginUserSpecFlowId":"831bb716-6ea7-41a8-af7b-aa8a18549ea4","CLIUserSpecFlowId":null,"ExecutionResults":[{"ContextType":"Scenario","FeatureFolderPath":"Features/Games","FeatureTitle":"Test","ScenarioTitle":"Test1","ScenarioArguments":[],"Status":"OK","StepResults":[{"Duration":"00:00:03.6108962","Status":"OK","Error":null,"Outputs":null},{"Duration":"00:00:02.8972892","Status":"OK","Error":null,"Outputs":null},{"Duration":"00:02:01.6962209","Status":"OK","Error":null,"Outputs":null},{"Duration":"00:00:00.0247738","Status":"OK","Error":null,"Outputs":null}],"Outputs":null}],"StepReports":null}

Steps to Reproduce

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
  <Settings projectName="NetmarbleWebQATeam" />
  <VSTest testRetryResults="Unified" passRateRelative="10%"/>
    <Execution stopAfterFailures="0" testThreadCount="Max" testSchedulingMode="Sequential" apartmentState="MTA" retryFor="Failing" retryCount="3" />

    <Environment testThreadIsolation="SharedAppDomain" platform="x86"/>
  <TestAssemblyPaths>
    <TestAssemblyPath>Team.dll</TestAssemblyPath>
  </TestAssemblyPaths>

    <Targets>

        <Target name="Chrome">
            <Filter>Browser_Chrome</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="Test_Browser" value="Chrome" />
            </DeploymentTransformationSteps>
        </Target>

    </Targets>

</TestProfile>

Run 2 or more tests and check TestExecution.json in bin folder

Link to Repro Project

No response

namilkimfree commented 2 years ago

find document https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Setup-the-LivingDocPlugin.html

github-actions[bot] commented 2 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.