Aborted tests

[johannesegger]

We have a project with about a thousand tests (many of them are data-driven tests). We recently updated to v2.0.5. When we now run all tests of this project, the run mostly aborts and the remaining tests are marked as being inconclusive. The number of executed tests until the abort happens varies. Did you ever experience this or do you have an idea what could cause this? When using the xUnit command line runner it works as expected. Some of the tests take quite a long time (up to five minutes), so could it be a timeout?

[citizenmatt]

There could be a couple of possible reasons. Firstly, make sure you're also running 2.0.0 RTM. There have been breaking changes during development of xunit 2, and it can cause the runner to fail in unexpected ways.

That said, I know that the runner can be a bit flaky under certain circumstances. I've only seen it happen with extensions such as xbehave that weren't following the normal sequence of runner progress messages, but there have been reports of it failing as you've described.

I've just rewritten the way the runner handles messages which should make it more robust. I should be releasing it in the next day or two - I'm trying to get a few other things into the release while I'm at it (ReSharper 9.1 support, annotations, etc.). I'll let you know when it's released.

[johannesegger]

Thanks for the quick response.

We are running the stable version of xUnit 2.0.0 at the moment. What do you mean with "... that weren't following the normal sequence of runner progress messages". What messages do you mean?

[citizenmatt]

xbehave extends xunit, with new syntax and commands that can run tests. As it's running tests, xunit reports progress messages to the runners. There was an issue with xbehave (since fixed) where it wasn't reporting the messages in the correct order, and that could cause issues with what the ReSharper runner was expecting. The recent changes to the ReSharper runner make it much more robust in these kinds of situations, and it should help with similar issues seen when not using xbehave or related extensions to xunit.

[johannesegger]

Ok I see, thanks for the explanation. We don't use xbehave, so we're curious if your changes will solve our issue. We're looking forward to it anyway, thanks for the great work.

[johannesegger]

I just tried version 2.0.6 and it still aborts when running the tests. I can't update to version 2.1.6, is this for the EAP?

[citizenmatt]

No, 2.1.6 is for ReSharper 9.1. The 2.0.6 release is for 9.0, but is essentially the same codebase. I'm intending to release new versions just for 9.1 for now, as it's a free upgrade from 9.0 to 9.1.

Can you get some logs for me, please? It's a bit of a multi-step process, but hopefully it'll show what's going wrong (alternatively, if you have a repro project you can send my way, that'd also help).

• Start Visual Studio with devenv /ReSharper.Internal
• Go to ReSharper → Options → Unit Testing and check "Enable Logging"
• Make sure the %TEMP%\JetLogs folder exists
• Run your tests

There should now be a log file in %TEMP%\JetLogs can you attach it, or post it to a gist or something? (I don't think there's anything personally identifying in there, or that would show anything useful about your code other than type names and namespaces, but it might be worth checking the file first)

[johannesegger]

I currently can't provide you a repro project as I'm doing super-duper secret stuff, but you can find a log file here (thanks for the instructions how to create one).

I also tried running only a subset of the tests and that always worked.

[citizenmatt]

Brilliant, thanks. With that test run, do you know which tests were aborted?

[johannesegger]

The red hand is only at Weingartner.Numerics.Spec.Pump.MillingHeadCreateScrewToolPathSpec.ShouldCreateCorrectToolPath. And some of the tests have the blue question mark.

[johannesegger]

Btw. this is a data-driven test with 12 cases. When I then ran this specific test again, it worked (log).

[citizenmatt]

Does it run ok when you run the assembly with the console runner? I'm seeing that xunit is sending the "test starting" message, but nothing else turns up.

Also, try disabling parallelisation, either by the CollectionBehaviour attribute (http://xunit.github.io/docs/running-tests-in-parallel.html) or via config (http://xunit.github.io/docs/configuration-files.html)

[johannesegger]

You mean the standard xUnit console runner? Yeah, I tried that and it worked. I'll try the other suggestions immediately.

[johannesegger]

Ok I disabled parallelization and now it aborted at a different test case (a data-driven test again). Do you need the log?

[citizenmatt]

Yes please, that would be helpful.

[johannesegger]

https://gist.github.com/eggapauli/47e83676092e93b79f08

[citizenmatt]

What test failed, btw?

[johannesegger]

Ohh sorry, I forgot: Weingartner.Numerics.Spec.Pump.CycloidFlankSolverSpec.derivative_should_be_negative_and_vertical_for_cusp_at_phi_is_zero, but just the one where Lobes == 7, the others succeeded.

[citizenmatt]

I'm at a bit of a loss, I'm afraid. The second log shows that xunit's giving me the right messages, and I'm sending the right messages to ReSharper, so there shouldn't be an aborted test there.

Is it consistent? As in, does running everything again cause the same test to fail, or is it random/intermittent?

The one thing that might be interesting is that it looks like you're using a custom implementation of ToString on your parameter class - it looks like a JSON string, and xunit's default serialisation is different. It might be worth commenting out the ToString method and seeing if that makes a difference.

[johannesegger]

I tried to comment the ToString method to use xUnit's default serialization, but that didn't help.

Currently it seems that the test runs always abort on a data-driven test which takes a non-simple data type as argument, but even the test case within a data-driven test varies.

[johannesegger]

So, we understand that you can't investigate on this as long as we can't provide you with a repro project. We hope to have time for this soon and we'll then come back to you. Thanks for the great support so far.

[citizenmatt]

Yes, I'm afraid I can't reproduce this, and can't see what's going wrong. A repro project would be ideal… Thanks!

[johannesegger]

Sorry we currently don't have time to investigate on this so I'm closing this for now. I may reopen this as soon as we can reproduce our issue. Thanks for the support so far.