R# fails to support Xunit.Combinatorial

[mwpowellhtx]

I get this exception. I figure this is most likely a R# issue, but in the event that it may be Xunit-based, thought I should file it here as well.

2018.03.21 14:36:53.987   ERROR JetBrains.Util.InternalErrorException: Unexpected: Unable to find any matching test cases
   at JetBrains.ReSharper.UnitTestRunner.Xunit.TestRunner.Run(XunitTestAssemblyTask assemblyTask)

2018.03.21 14:36:54.002    WARN Element <Path.To.Tests> was left pending after its run completion.
2018.03.21 14:36:54.002    WARN Element <Path.To.Tests.Verification> was left pending after its run completion.

[mwpowellhtx]

I should also clarify, I am extending CombinatorialValuesAttribute with custom ranges.

One thing I have discovered, coming from an NUnit perspective, at any rate, is that (re-)evaluating the ranges of values constantly is a SIGNIFICANT performance hit.

So, what I like to do is statically initialize my ranges ONCE, especially for more complex ranges. This seems to improve performance SIGNIFICANTLY.

That being said, I don't know how Xunit and/or Xunit.Combinatorial are evaluating the parameters. In other words, are you looking for CombinatorialValuesAttribute only? Or Attributes including those derived from such?

[mwpowellhtx]

My suspicion is that you are looking for only CombinatorialValuesAttribute.

Edit: This is likely the root cause of the issue, exposed via R# is tangential, in my opinion. When this part is resolved, perhaps then test cases will be discovered, etc, etc, et al?

[mwpowellhtx]

@AArnott Any word on this issue? Is this a VS2017 solution? I think it is critical for custom user-provided combinatorial ranges.

[AArnott]

Is this a VS2017 solution?

I'm not sure what you're asking or how the VS version the solution is defined in relates to this feature request. The source code in this repo is only expected to load in VS 2017 because it uses the new MSBuild SDK project file syntax.

I wouldn't object to searching for derivative types of the CombinatorialValuesAttribute. Would you care to send a PR? I personally am not likely to get to this in the next several weeks.

[mwpowellhtx]

@AArnott Short of submitting a PR to the repo, here's the solution I think will work:

internal static IEnumerable<object> GetValuesFor(ParameterInfo parameter)
{
    var valuesAttrib = parameter.GetCustomAttributes(true)
        .SingleOrDefault(a => a is CombinatorialValuesAttribute);

    return valuesAttrib != null
        ? ((CombinatorialValuesAttribute) valuesAttrib).Values
        : GetValuesFor(parameter.ParameterType);
}

At least it's a start, without possibly needing to examine the MethodInfo signature, etc.

[mwpowellhtx]

@AArnott I'm sure you've got plenty on your plate in your day job; however, would you mind providing some idea as to a time frame? I've got some work going on depending on XUnit Combinatorial support. Thanks!

[AArnott]

If your primary concern is the derivability of this attribute, isn't this a dupe of #13?

[AArnott]

Let me know if you think it's not a dupe.

[mwpowellhtx]

Along these lines, yes. Bottom line, it still isn't fixed. I've proposed the solution which I think should work. What's wrong with that approach rather than looking for just CombinatorialValuesAttribute?

[mwpowellhtx]

@AArnott To be clear, this does not address the issue at hand:

public class MyValuesAttribute : CombinatorialDataAttribute { /* ... */ }

And which in your test case...

// ...
var attrib = new MyValuesAttribute();
// ...

Make sense? Is that clear, now?

[AArnott]

If you take a look at the sample I added as well, you'll see that it doesn't do anything special other than what you describe. And the fact that it works in the VS 2017 Test Explorer satisfies me that either the issue is you're using VS 2015 or the R# test runner instead of the VS one.