JoshKeegan / xRetry

Retry running tests via Xunit and Specflow
MIT License
47 stars 14 forks source link

Error: "Test methods cannot have multiple Fact or Theory attributes" #74

Closed dluc closed 3 years ago

dluc commented 3 years ago

Any suggestion about merging two or more attributes?

e.g. the following code doesn't compile because "Test methods cannot have multiple Fact or Theory attributes"

[SkippableFact] [RetryFact]
public void SomeTest()
{
    // some code
}
CmdrRamon commented 3 years ago

it's because SkippableFact and RetryFact inherit from FactAttribute so you need to remove SkippableFact

JoshKeegan commented 3 years ago

In xunit there can only be one attribute denoting that a method is a test (natively that's Fact and Theory but things like RetryFact and SkippableFact follow the same pattern) and internally that attribute tells xunit which "discoverer" to go and run to generate a test case to actually execute. These test cases (or a runner that they call out to) is what contain the logic to do things like retries or skipping at runtime.

So if we could have two attributes on a single method, xunit would need to know how to combine the two different test cases that each attribute indicates should be generated into one test case that it can actually run. xunit has no way of knowing how to do this out of the box, but you could do it with code if you really need it by writing your own RetrySkippableFactAttribute , RetrySkippableFactDiscoverer, RetrySkippableTestCase and RetrySkippableTestCaseRunner. That would be no simple task though and would be overkill unless you wanted to have this as a standard pattern available to be used on a large number of tests.

What I would recommend is picking one of the attributes and reworking the test to not need the other. e.g. using a RetryTheory with MemberData to generate test cases at runtime could allow for cases to be skipped by not returning them from the function that generates the test data. There's an example of a retry with memder data at https://github.com/JoshKeegan/xRetry/blob/master/test/UnitTests/Theories/RetryTheoryNonSerializableDataTests.cs Or alternatively if that wouldn't work you could write the retry logic within the test itself.

dluc commented 3 years ago

Thanks for the feedback. Unfortunately we need both features. The ability to decorate tests with attributes to add cross cutting features is a great thing, I hope future versions of the test framework will enable my scenario too. Looks like we might have to write a custom attribute (e.g. RetrySkippableFactAttribute), combining the source code of the two.

JoshKeegan commented 3 years ago

Fair enough. If you do write a library (and then also decide to open source it), please come back & leave a link here so that anyone stumbing accross this in the future can use it :)

JoshKeegan commented 2 years ago

@dluc I've started work on functionality to skip tests dynamically at runtime and have the xunit side of things working in a branch. Specflow support is still WIP and I'll finish that before releasing, but look out for the next release!