search

reportportal / agent-net-specflow

Report Portal agent for SpecFlow
Apache License 2.0
10 stars 4 forks source link

Retry not working correctly using vstest + specflow agent #85

Closed heilwood closed 1 month ago

heilwood commented 1 month ago

We are using retry mechanism in our azure pipelines with vstest task which should retry failed tests, but is not working correctly with specflow agent, instead of retrying failed tests vstest re-running all available tests and problems looks like in naming of tests using specflow agent, vstest selecting incorrect filter for specific test, instead of using name of test scenario + test scenario example, he using just feature name.

image

Output from azure pipeline (Check Test Case Filter):

Failed tests: 1; Total tests: 20; Total failed tests as returned by TCM: 1. Found 1 filtered test case sources. **************** Rerunning failed tests for Test run 2656682 ********************* Max attempts: 2; Current attempt: 1; Failed test cases threshold: 30; Failed test cases max limit: 0 C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@C:\agent\_work\_temp\jlbvhts0s42.tmp" Microsoft (R) Test Execution Command Line Tool Version 17.7.2 (x64) Copyright (c) Microsoft Corporation. All rights reserved. vstest.console.exe /TestCaseFilter:"(FullyQualifiedName=Automation.Common.Addons.#()::Target:Chrome_PC_Functional/TestAssembly:Automation.Price)"

Packages version: image

Test Runner: SpecRun

nvborisenko commented 1 month ago

ReportPortal is not responsible for running tests, it is reporting test results. We cannot fix it.

I guess you should post new issue for VsTest repo or SpecRun.

heilwood commented 1 month ago

ReportPortal is not responsible for running tests, it is reporting test results. We cannot fix it.

I guess you should post new issue for VsTest repo or SpecRun.

VStest re-running failed tests based on test name which present in logs, SpecFlow agent not responsible for logging? It can be that SpecFlow Agent logging incorrect test names?

nvborisenko commented 1 month ago

VStest re-running failed tests based on test name which present in logs

I hope it is just an assumption.

SpecFlow agent not responsible for logging?

RP.SpecFlow does log nothing.

It can be that SpecFlow Agent logging incorrect test names?

It logs nothing.

@heilwood please uninstall all RP NuGet packages, and if the issue remains reproducible - we dive deeper.

heilwood commented 1 month ago

you are right, problem is on runner side somewhere, thank you for your effort!