coverlet-coverage / coverlet

Cross platform code coverage for .NET
MIT License
2.99k stars 387 forks source link

Holding a thread longer open then a test runs through causes coverlet to problably crash internaly and dont calc code coverage #1678

Open ReneKochITTitans opened 3 months ago

ReneKochITTitans commented 3 months ago

Hi,

yesterday I opened https://github.com/coverlet-coverage/coverlet/issues/1677, which i further investigated today, because the title + text are complelty different I choosed to close the old one and open a new one with a bunch of more details. Today I was able to pinpoint the bug (maybe even not even a bug, maybe a known behaviour):

When I open in a thread that needs longer then a test-run is, the thread blocking coverlet from getting any results for the complete project in it (not only the class or the test the failure code is in).

I was able to write a reproduce:

`public class TestServiceA : ITestServiceA { public string DoubleText(string str) { new Thread(async () => { try { Thread.Sleep(10); } catch (Exception e) { } }).Start();

    return str + str;
}

public string TrippleText(string str)
{
    return str + str + str;
}

}`

`namespace EmailService.Services;

public interface ITestServiceB { }

public class TestServiceB(ITestServiceA testServiceA) : ITestServiceB { public string Cut(string str) { var newStr = testServiceA.DoubleText(str); return newStr.Substring(0,5); }

public string CutSmall(string str)
{
    var newStr = testServiceA.DoubleText(str);
    return newStr.Substring(0,4);
}

}`

` [Test] public void Rep() { var serviceA = new TestServiceA(); var result = serviceA.DoubleText("Test"); Assert.That(result, Is.EqualTo("TestTest")); }

    [Test]
    public void Rep2()
    {
           var serviceA = new TestServiceA();

        var serviceB = new TestServiceB(serviceA);

        var result = serviceB.CutSmall("Test");
        Assert.That(result, Is.EqualTo("Test"));
    }`

So what we have here is a Thread within the code that should be tested. The code above produces the code coverage just fine. If we now start adding more ms to the Thread.Sleep until a point where the test execution is faster then the Thread opened, the coverage for ALL tests in the given test project is 0,0% (e.g. 1000ms)

There is no errorcode or whatever, it just seems that the code coverage calc crashes internaly (?).

In my real life usecase when I found this bug, I have 15 tests are with 0% coverage because I wait 1second to long in the thread. If I remove the Thread.Sleep or reduce it, all 15 tests will be having code coverage caluclated, even only 1 out of the 15 tests using the method where the thread is in.

Bertk commented 2 months ago

Thank you for bringing this up to our attention. Please provide more information which allows us to reduce the scope of this problem.

Configuration (please complete the following information): Please provide more information on your .NET configuration:

  • Which coverlet package and version was used? [coverlet.collector | coverlet.msbuild | coverlet.console]
  • Which SDK version of .NET is the code running on? [8.0.? | 7.0.? | 6.0.?]
  • What OS and version, and what distro if applicable?
  • What is the architecture (x64, x86, ARM, ARM64)?

Additional context Add any other context about the problem here.

:exclamation: Please also read Known Issues

You can also enable diagnostic logs which will provide additional information (Please see https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration).