Inconsistent test results

[FeodorFitsner]

Discussion: http://help.appveyor.com/discussions/problems/6117-some-unit-tests-are-skippednot-finished Affected project: https://ci.appveyor.com/project/Takumii/markify/history

[FeodorFitsner]

I did some investigation. At first, I thought it's the same issue when AppVeyor build agent was unable to post all test results to https://ci.appveyor.com service. However, I enabled RDP and found out that there were no errors sending test results to AppVeyor UI.

Then I added counters for added/updated tests into build agent to see how many tests are submitted by xUnit to AppVeyor Build Worker API. Those counters are incremented every time POST/PUT api/tests or POST/PUT api/tests/batch endpoints called. It looks like it's xUnit runner is sending varying number of test results to AppVeyor build agent. You can see counter results in these three consequent builds: https://staging-ci.appveyor.com/project/FeodorFitsnerStaging/markify/build/1.0.21#L1026 https://staging-ci.appveyor.com/project/FeodorFitsnerStaging/markify/build/1.0.22#L1019 https://staging-ci.appveyor.com/project/FeodorFitsnerStaging/markify/build/1.0.23#L1019

Could it be that xUnit runner is failing to call AppVeyor API but not reporting the error to a console? I see that tests are wrapped with OpenCover - could it the issue? I tried running those tests without OpenCover, but they are failing with memory-related exceptions: https://staging-ci.appveyor.com/project/FeodorFitsnerStaging/markify/build/1.0.25#L900 Is it possible to run those tests without OpenCover, what's correct way?

@onovotny - do you have any idea what it might be? Is there any way to debug/troubleshoot those tests?

[Julien-Pires]

It seems that OpenCover isn't innocent. I've tried a build with no code coverage and every tests have passed successfully. Here is the build result : https://ci.appveyor.com/project/Takumii/markify/build/1.0.407/job/3f389xslt46v62yw And here the branch with OpenCover disabled : https://github.com/Julien-Pires/Markify/tree/no-code-coverage

You can see that 539 tests has passed contrary to around 250 on build with OpenCover enabled. Here is the latest build I've made with OpenCover enabled : https://ci.appveyor.com/project/Takumii/markify/build/1.0.408.

It could be a change in my own code that OpenCover doesn't like. The test project that get stuck is always the same and it's an F# project. Probably OpenCover doesn't support every aspect of the code generated by the F# compiler. Probably a corner case where OpenCover gets stuck in some infinite loop.

The first time I noticed this issue was after this change : https://ci.appveyor.com/project/Takumii/markify/build/1.0.381.

I should probably try to run OpenCover on local. Currently, I run the code coverage only on the CI.

[FeodorFitsner]

OK, thanks for the update! I think it makes sense to close this issue then as it's more related to OpenCover.

[Julien-Pires]

I've tried running OpenCover on my computer and it runs successfully. Nothing gets stucked. Before testing I was enough confident that it would be a bug from OpenCover but now I'm not completely sure.

The test project that gets stuck on AppVeyor is a project that reads files to get some data. Could it be a problem with some files that get locked and the test is waiting for the lock to be open?

[FeodorFitsner]

Maybe 2 CPU cores on VM is an issue? I'm sure you are running it locally on 4+ cores. You may try provisioning a VM with similar specs on, say, Azure and try running there to see if it behaves differently from your local machine.

[Julien-Pires]

Similar spec as AppVeyor with only 2 cores or similar to my computer which is 4+ cores?

[FeodorFitsner]

Similar to AppVeyor with 2 cores.

[christianspecht]

Same here - I have a similar issue now.

My project is a .NET Core console application with xunit.net tests (unit and integration tests in separate projects). No coverage tool, just xunit.

Until Feb 24, the number of tests in the "Tests" tab always matched with the number displayed by my build script in the console.

Here's the last "successful" one:
https://ci.appveyor.com/project/ChristianSpecht/scm-backup/build/0.1.43 The "Tests" tab shows 98 tests, and the console as well (71 unit tests + 27 integration tests = 98):

=== TEST EXECUTION SUMMARY ===
   ScmBackup.Tests  Total: 71, Errors: 0, Failed: 0, Skipped: 0, Time: 0.308s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.
 [...]
=== TEST EXECUTION SUMMARY ===
   ScmBackup.Tests.Integration  Total: 27, Errors: 0, Failed: 0, Skipped: 0, Time: 2.694s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

---

Then I didn't work on my project for six or seven weeks, and today I upgraded it to Visual Studio 2017 and .NET Core 1.1:
https://ci.appveyor.com/project/ChristianSpecht/scm-backup/build/0.1.44

In the "Console" tab, you can see that the same 98 (71 + 27) tests were actually executed:

Total tests: 71. Passed: 71. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 5.0089 Seconds
[...]
Total tests: 27. Passed: 27. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 5.4599 Seconds

But the "Tests" tab says "29", and when I click on it, it shows a list of 29 tests (all 27 integration tests and 2 of the unit tests, marked yellow).

---

Then I read in your documentation that the default image is VS 2015 and I have to specify the VS 2017 image, so I changed that and pushed again:

https://ci.appveyor.com/project/ChristianSpecht/scm-backup/build/0.1.45

Nearly the same result as before: 71 + 27 tests executed in the console, and this time 27 in the "Tests" tab (all of the integration tests, none of the unit tests).

[FeodorFitsner]

Will take a look.