Specflow.xRetry and IUnitTestRuntimeProvider.TestIgnore is failing the test

[piotrhabecki]

Hello,

When I am using xRetry mechanism for spec flow and I want to custom ignore test cases with the usage of TestIgnore from IUnitTestRuntimeProvider the xRetry is causing the output of the test to fail not to be ignored.

Example:

@retry(3)
Scenario Outline: Example of failing test with retry and ignore
Given I will try to use the calculator
And The current value is <ForbiddenValue>
And I will add numbers 1 and 2
Then result is 3

Examples:
|ForbiddenValue|
|5|

Implementation of step: And the current value is

        [Given(@" the current value is (.*)")]
        public void GivenTheCurrentValueIs(int forbiddenValue)
        {
            if (ForbiddenValue== 5)
                unitTestRuntimeProvider.TestIgnore($"This scenario should not be ran");
        }

This set of usage will fail the test case instead of ignoring the test case.

[JoshKeegan]

Thanks for raising the issue! The short answer is that it seems like this is a missing feature and I'll need to do work to add suport for it in a future release.

Long answer: Specflow make use of another library that changes the behaviour of xunit test execution Xunit.SkippableFact, but unfortunately xUnit can only have one attribute per fact or theory to determine how a test is run (e.g. [Fact] would be the native one, [RetryFact] would be the one from this library for running with retries). So we can't have two different libraries in charge of how a test runs (better/more detailed explanation in #74). When we generate the xUnit code behind each Specflow scenario that means that if we add retries we have to use [RetryFact] and not [SkippableFact] like native Specflow would generate. Internally within Specflow when you call IUnitTestRuntimeProvider.TestIgnore for xUnit this runs Skip.If(... which the [RetryFact] does not understand, and since Skip.If throws an exception the test ends up failing rather than being skipped.

To fix this, I'll need to add similar skip functionality to xRetry and then we can register a different implementation of IUnitTestRuntimeProvider with BoDi so that Specflow uses our skip implementation instead.

This will be a large change though and I don't currently have time to work on it. Hopefully in a few months I have time and can come back to this.

[JoshKeegan]

This is fixed with the release of v1.7.0, please try it out and let me know if you have any issues.

[piotrhabecki]

Thank You Josh. You are my hero ;-)