Kover thinks there are 7 million branches

[NikolayMetchev]

Describe the bug I have attached a source file which for some reason seems to think there are 7 million branches causing our code coverage numbers to be very low. I have also attached the xml file produced by the kover tool. Jacoco produces the same problem which makes me think this is a kotlin problem but I didn't know where to file it. Here is the relevant section from the xml file:

<method name="invokeSuspend" desc="(Ljava/lang/Object;)Ljava/lang/Object;">
<counter type="INSTRUCTION" missed="0" covered="48"/>
<counter type="BRANCH" missed="7615047" covered="0"/>
<counter type="LINE" missed="0" covered="7"/>
</method>
<counter type="INSTRUCTION" missed="0" covered="48"/>
<counter type="BRANCH" missed="7615047" covered="0"/>
<counter type="LINE" missed="0" covered="7"/>
<counter type="METHOD" missed="0" covered="1"/>
</class>

Errors If present, stacktraces or files from build/kover/errors directory

Expected behavior There should be far less than 7 million branches

Reproducer Not sure how to reproduce for a small case. Hopefully the code and xml file give you some pointers.

Reports If applicable, report files or screenshots.

Environment

• Kover Gradle Plugin version: 0.6.1
• Gradle version: 7.5.1
• Kotlin project type: Kotlin/JVM
• Coverage Engine version (if customized in build script): Default
• Other context important for this bug: Linux LedgerRoutes.kt.gz report.xml.gz

[zuevmaxim]

@NikolayMetchev Could you please share com/paxos/metalsapi/routes/LedgerRoutes$addRoutes$1$39.class file?

[NikolayMetchev]

@zuevmaxim I have shared it. it is in LedgerRoutes.kt.gz

[NikolayMetchev]

oh you mean the actual class file? sorry bear with me.

[NikolayMetchev]

LedgerRoutes.tar.gz

[shanshin]

@NikolayMetchev, may you also send the contents of the directory build/kover? Binary coverage reports are located there, for each test task there should be its own .ic file

[NikolayMetchev]

kover.zip

[NikolayMetchev]

@shanshin you tagged the issue as confirmed. Can you clarify what the problem is? Is there possibly a workaround? What are the chances of a fix?

[shanshin]

@NikolayMetchev, yes, the problem is really present, at the moment we are trying to find the cause

[shanshin]

@NikolayMetchev, after the build, is the build/kover/errors directory empty in the project where the incorrect report is generated?

[shanshin]

Is it possible that in your test the code inside post("/ledger/advices/process/{id}") has been called more than 2 billion times?

[NikolayMetchev]

Hi, there isn't a directory build/kover/errors at all. That endpoint is not called 2 billion times, at most it is a few hundred.

[NikolayMetchev]

Yeah, I checked. It is called roughtly 10-12 times. Using io.ktor.server.testing.TestApplicationEngine

[zuevmaxim]

@NikolayMetchev Could you please try a new version of the coverage agent?

for Kotlin Gradle script

kover {
    engine.set(kotlinx.kover.api.IntellijEngine("1.0.689"))
}

for Groovy

kover {
    engine = kotlinx.kover.api.IntellijEngine("1.0.689")
}

If the issue persists, please share the content of build/kover and build/kover/errors directories

[NikolayMetchev]

Hi I am using the groovy version, but for some reason I can't get it to work. I get the following error:

kover {
    engine = kotlinx.kover.api.IntellijEngine(libs.versions.intellijCoverageAgent.get())
}

• What went wrong: A problem occurred evaluating root project 'metals'.

  Could not get unknown property 'kotlinx' for extension 'kover' of type kotlinx.kover.api.KoverProjectConfig.

[NikolayMetchev]

I think the problem is that some parts are internal such as CoverageEngineVendor

[shanshin]

@NikolayMetchev, try

kover {
    engine = new kotlinx.kover.api.IntellijEngine("1.0.689")
} 

[NikolayMetchev]

we look to have made progress. Down to 2 million from 7.:

<method name="invokeSuspend" desc="(Ljava/lang/Object;)Ljava/lang/Object;">
<counter type="INSTRUCTION" missed="0" covered="48"/>
<counter type="BRANCH" missed="2136016" covered="0"/>
<counter type="LINE" missed="0" covered="7"/>
</method>
<counter type="INSTRUCTION" missed="0" covered="48"/>
<counter type="BRANCH" missed="2136016" covered="0"/>
<counter type="LINE" missed="0" covered="7"/>
<counter type="METHOD" missed="0" covered="1"/>
</class>```

[NikolayMetchev]

no errors

[zuevmaxim]

@NikolayMetchev could you please share the content of build/kover ? I need .ic file from there

[NikolayMetchev]

kover (1).zip

[zuevmaxim]

@NikolayMetchev Thank you for the attached files! I have analyzed them, and I see that after the tests run, the coverage report already includes an enormous number of branches, but there just a few in the byte code. Still, I cannot understand the reason for it, and cannot reproduce the issue with the attached class files. So, without a reproducer, it is unclear where the bug comes from.

Is your project open source? If so, could you please share a link to it?

If not, we would be happy to get some more clues from you:

• Does your project use any framework or library manipulating with the byte code?
• Could you please share the XML report generated with the JaCoCo engine? (you noted that it also has problems)
• If you can extract a reproducer (of any size), it would be great

[NikolayMetchev]

Hi, It seems there was an error file. I will spend some time today trying to create a reproducer. Maybe this log file will help you figure out what is wrong? test.log

[NikolayMetchev]

Here some more information. I just tried to reproduce locally (M1 Mac). In order to do that I had to ignore some tests as they only pass in CI due to locale issues. In CI we use linux debian and Intel CPUs. If I filter locally to just a few tests that actually call the function in question the problem doesn't reproduce. If I run the entire test suite locally a different class has a problem of much smaller magnitude see below. I have attached the test.ic and report.xml from the local run. The error file is attached in the previous comment. I will now try and run with Jacoco and see if the same thing happens. This feels like there is a race condition somewhere which will make it very hard for me to make a small reproducer.

[report.xml.gz](https://github.com/Kotlin/kotlinx-kover/files/10112385/report.xml.gz)

[test.ic.gz](https://github.com/Kotlin/kotlinx-kover/files/10112381/test.ic.gz)

<class name="com/paxos/metalsapi/routes/LedgerRoutes$addRoutes$1$38" sourcefilename="LedgerRoutes.kt">
<method name="invokeSuspend" desc="(Ljava/lang/Object;)Ljava/lang/Object;">
<counter type="INSTRUCTION" missed="0" covered="56"/>
<counter type="BRANCH" missed="27450" covered="0"/>
<counter type="LINE" missed="0" covered="7"/>
</method>
<counter type="INSTRUCTION" missed="0" covered="56"/>
<counter type="BRANCH" missed="27450" covered="0"/>
<counter type="LINE" missed="0" covered="7"/>
<counter type="METHOD" missed="0" covered="1"/>
</class>

[zuevmaxim]

@NikolayMetchev Do you have tests running in a fork? maxParallelForks or forkEvery options?

[NikolayMetchev]

Yes I do:

    maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1

[NikolayMetchev]

As a workaround I am trying to filter that class from the xml report but no incantation seems to be working. Here is my latest attempt:

kover {
    filters {
        classes {
            excludes.addAll('com.paxos.metalsapi.routes.LedgerRoutes.addRoutes.1.*')
        }
    }
    xmlReport {
        overrideFilters {
            classes {
                excludes.addAll('com.paxos.metalsapi.routes.LedgerRoutes.addRoutes.1.*')
            }
        }
    }
}

Please can you tell me if it is possible to exclude that class from the xml report?

[zuevmaxim]

I suppose it should be com.paxos.metalsapi.routes.LedgerRoutes\$addRoutes\$1\$*

[NikolayMetchev]

Thank you that worked!

[NikolayMetchev]

One annoying thing is that changing the filter causes the tests to be re-run (cache miss in gradle). Is this intended behaviour?

[shanshin]

One annoying thing is that changing the filter causes the tests to be re-run (cache miss in gradle). Is this intended behaviour?

This is expected behavior, because in version 0.6.1, changing the report filters also changes the instrumentation filter of the classes being run, so you need to rerun the test to collect coverage of only the classes being instrumented.

[NikolayMetchev]

Good to know. Thanks for the clarification.

[NikolayMetchev]

Perhaps you need to add logging as a feature so we can debug?

[zuevmaxim]

@NikolayMetchev Could you please try to rerun with maxParallelForks=1 ? And upload a file with errors

[shanshin]

@NikolayMetchev, is the problem still reproducible on the 1.0.709 version?

[NikolayMetchev]

sorry about not responding earlier. It went off my radar. Will try running with the latest version shortly. (I will need to reproduce first to make sure I have it correct again).

[NikolayMetchev]

The issue doesn't seem to reproduce any longer with version 1.0.689. A few things have changed since the last time I tested it:

Gradle 7.5.1 Kotlin 1.8.0

I will now try it with 1.0.709 to see if that behaves differently

[NikolayMetchev]

Yep, looks like it may get fixed by upgrading Gradle, kotiln or possibly just changing the code. I can't be sure what has fixed this issue but I can't reproduce it anymore.

[shanshin]

Thanks!

Feel free to open new bug issue if the problem reproduced again.