VSTest testRetryResults="Unified" not working via AzureDevops builds

[lharriespipe]

SpecFlow Version

3.9.8

Which test runner are you using?

SpecFlow+ Runner

Test Runner Version Number

3.9.7

.NET Implementation

.NET Core 3.1

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK

SpecFlow Section in app.config or content of specflow.json

{ "bindingCulture": { "language" :"en-us" }, "language": { "feature": "en-us" } }

Issue Description

Adding the below config into the profile shows random failures as passing in Visual studio test explorer however a random failure fails the build in azure devops

<VSTest testRetryResults="Unified" passRateAbsolute="1"/>

Is there a way to get this behaviour so our builds can report as passed if we encounter a random failure?

Steps to Reproduce

1. Add <VSTest testRetryResults="Unified" passRateAbsolute="1"/> to srprofile
2. run in VS and observe random failures are displayed as passing
3. run in a build and observe build fails if it encounters a random failure

Link to Repro Project

No response

[SabotageAndi]

is the Default.srProfile copied to the output directory? Is the file really named Default.srProfile? VS likes to create the file as Default1.srProfile On which OS are you? Linux is case sensitive, so check the file name again.

[lharriespipe]

All profiles are copied to the output directory. We have multiple profiles for different environments and use a .runsettings file to point at the relevant .srprofile, based on the tests performed it is definitely picking up the correct .srprofile. I am using windows locally and our build agents are also running windows.

[SabotageAndi]

As we stopped maintaining the SpecFlow+ Runner (https://specflow.org/using-specflow/the-retirement-of-specflow-runner/) I am closing this issue.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.