search

JoshKeegan / xRetry

Retry running tests via Xunit and Specflow
MIT License
45 stars 14 forks source link

Consider the test passed if it passed on one of the retry attempts #173

Open dsdilpreet opened 1 year ago

dsdilpreet commented 1 year ago

Hi Josh,

Great work with the retry package, it really makes the life easier!

I have a question: I am using Azure DevOps Specflow + LivingDoc with xUnit runner and xRetry package for automation testing. Everything is working and I can see the test results being published in LivingDoc. My issue is that if the test failed on first attempt but passed on an subsequent attempt, it is being reported as failed test but I want it to get reported as passed test. Do you know if there is any config / setting which I can change to change the behaviour? If there isn't any setting, perhaps this could be a good feature if its possible?

Thanks,

JoshKeegan commented 1 year ago

Hi @dsdilpreet, I've been able to reproduce this, but I don't think there is anything I am able to do to fix it. It appears to be an issue with the SpecFlow.Plus.LivingDocPlugin nuget package. That incorrectly marks the test as failed in the TestExecution.json file it produces. Can you raise this issue with the SpecFlow team?

Since that package is closed source I don't know for sure how it works internally, but I expected it's getting its test results from SpecFlow directly rather than the test runner. Thanks, Josh

dsdilpreet commented 1 year ago

Hi Josh,

Thank you for looking into the issue. I have reported it to SpecFlow and waiting for their response now.

Dilpreet