search

hcoles / pitest

State of the art mutation testing system for the JVM
Apache License 2.0
1.68k stars 358 forks source link

interop problem with some kotlin inline methods #764

Closed christophsturm closed 6 months ago

christophsturm commented 4 years ago

if I run ./gradlew pitest on this branch of my project: https://github.com/christophsturm/r2dbcfun/tree/long-text , pitest fails with an exception:

~/Projects/mine/r2dbc-fun% ./gradlew clean  pitest

> Task :pitest FAILED
16:43:21 PIT >> INFO : ---------------------------------------------------------------------------
16:43:21 PIT >> INFO : Enabled (+) and disabled (-) features.
16:43:21 PIT >> INFO : -----------------------------------------
16:43:21 PIT >> INFO : +FANN           Filters mutations in classes and methods with matching annotations of class or runtime retention
16:43:21 PIT >> INFO :   [annotation]    Annotation to avoid (full package name not required)
16:43:21 PIT >> INFO : +FENUM          Filters mutations in enum constructors
16:43:21 PIT >> INFO : +FFBLOCK        Filters mutations in code duplicated by finally block inlining
16:43:21 PIT >> INFO : +FFEACH         Filters mutations in compiler generated code that implements for each loops
16:43:21 PIT >> INFO : +FFLOOP         Filters any mutations to increments in for loops as they may cause timeouts
16:43:21 PIT >> INFO : +FINFINC        Filters mutations to increments that may cause infinite loops
16:43:21 PIT >> INFO : +FINFIT         Filters mutations that may cause infinite loops by removing calls to iterator.next
16:43:21 PIT >> INFO : +FINULL         Filters mutations in compiler generated code that checks for null by calling getClass
16:43:21 PIT >> INFO : +FKOTLIN        Filters out junk mutations in bytecode created by compiler for kotlin language features
16:43:21 PIT >> INFO : +FLOGCALL       Filters mutations in code that makes calls to logging frameworks
16:43:21 PIT >> INFO : +FMRNULL        Filters mutations in compiler generated code that inserts Objects.requireNonNull for method references
16:43:21 PIT >> INFO : +FRETEQUIV      Filters return vals mutants with bytecode equivalent to the unmutated class
16:43:21 PIT >> INFO : +FSEQUIVEQUALS  Filters equivalent mutations that affect only performance in short cutting equals methods
16:43:21 PIT >> INFO : +FSTATI         Filters mutations in static initializers and code called only from them
16:43:21 PIT >> INFO : +FSTATINIT      Filters mutations in static initializers and code called only from them
16:43:21 PIT >> INFO : +FTRYWR         Filters mutations in code generated for try with resources statements
16:43:21 PIT >> INFO : -CLASSLIMIT     Limits the maximum number of mutations per class
16:43:21 PIT >> INFO :   [limit]         Integer value for maximum mutations to create per class
16:43:21 PIT >> INFO : -EXPORT         Exports mutants bytecode and other details to disk
16:43:21 PIT >> INFO : ---------------------------------------------------------------------------
16:43:21 PIT >> FINE : Running report with ReportOptions [targetClasses=[r2dbcfun.*], excludedMethods=[], excludedClasses=[], excludedTestClasses=[], codePaths=[/Users/christoph/Projects/mine/r2dbc-fun/build/classes/java/main, /Users/christoph/Projects/mine/r2dbc-fun/build/classes/kotlin/main], reportDir=/Users/christoph/Projects/mine/r2dbc-fun/build/reports/pitest, historyInputLocation=null, historyOutputLocation=null, sourceDirs=[/Users/christoph/Projects/mine/r2dbc-fun/src/main/resources, /Users/christoph/Projects/mine/r2dbc-fun/src/main/java, /Users/christoph/Projects/mine/r2dbc-fun/src/main/kotlin], classPathElements=[/Users/christoph/.gradle/caches/modules-2/files-2.1/org.pitest/pitest-junit5-plugin/0.12/6e1ee587a3457b0e7d636a5ce2e554fac62ec99/pitest-junit5-plugin-0.12.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.pitest/pitest/1.5.1/3914ec0629c931560e0124247c9221aa0cd1c6d/pitest-1.5.1.jar, /Users/christoph/Projects/mine/r2dbc-fun/build/classes/java/test, /Users/christoph/Projects/mine/r2dbc-fun/build/classes/kotlin/test, /Users/christoph/Projects/mine/r2dbc-fun/build/resources/test, /Users/christoph/Projects/mine/r2dbc-fun/build/classes/java/main, /Users/christoph/Projects/mine/r2dbc-fun/build/classes/kotlin/main, /Users/christoph/Projects/mine/r2dbc-fun/build/resources/main, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.strikt/strikt-core/0.25.0/ea965f14c7bc0573898fac145255dd8c2ec27b67/strikt-core-0.25.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/dev.minutest/minutest/1.11.0/712f219e05e5eae46556714ceac501bbffd75c17/minutest-1.11.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.christophsturm/filepeek/0.1.2/347482ee8f7b82e5dbc95ee0d61930e4fe4a5695/filepeek-0.1.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.3.72/916d54b9eb6442b615e6f1488978f551c0674720/kotlin-stdlib-jdk8-1.3.72.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.3.72/86613e1a669a701b0c660bfd2af4f82a7ae11fca/kotlin-reflect-1.3.72.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.r2dbc/r2dbc-h2/0.8.3.RELEASE/7a7469e68115e60531ed326f2673686847d47afa/r2dbc-h2-0.8.3.RELEASE.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.r2dbc/r2dbc-postgresql/0.8.2.RELEASE/524887193f7a95c674318a9b0d2b1205ef336ec0/r2dbc-postgresql-0.8.2.RELEASE.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.r2dbc/r2dbc-spi/0.8.1.RELEASE/17e0dbda7e0da3e8a570597509b876bacc776dd0/r2dbc-spi-0.8.1.RELEASE.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-reactive/1.3.5/e07ebba2d47ffe161c7b31ca0daf4792d7c482d6/kotlinx-coroutines-reactive-1.3.5.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.testcontainers/postgresql/1.14.1/3ee517af68a779ffca188fb2306163f94d69b347/postgresql-1.14.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.flywaydb/flyway-core/6.4.1/889ae0316ceadb713fb590b2278ff8fa8898e6aa/flyway-core-6.4.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-launcher/1.6.2/d866de2950859ca1c7996351d7b3d97428083cd0/junit-platform-launcher-1.6.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.2/1752cad2579e20c2b224602fe846fc660fb35805/junit-platform-engine-1.6.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.2/7644a14b329e76b5fe487628b50fb5eab6ba7d26/junit-platform-commons-1.6.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.6.2/c0833bd6de29dd77f8d071025b97b8b434308cd3/junit-jupiter-engine-5.6.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.2/c9ba885abfe975cda123bf6f8f0a69a1b46956d0/junit-jupiter-api-5.6.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-test/1.3.5/adb10783a05cab9d9e68a309ad0300de9eaa767a/kotlinx-coroutines-test-1.3.5.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.h2database/h2/1.4.200/f7533fe7cb8e99c87a43d325a77b4b678ad9031a/h2-1.4.200.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.postgresql/postgresql/42.2.12/1ed5b5f16a67f312a50d420e8bcb7d30b40b033d/postgresql-42.2.12.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.3.72/3adfc2f4ea4243e01204be8081fe63bde6b12815/kotlin-stdlib-jdk7-1.3.72.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-debug/1.3.5/756da7058ceeb644775ee943a2358fb17db50139/kotlinx-coroutines-debug-1.3.5.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core/1.3.5/b245208f2b844c02dbb506312d4859bde93bce8d/kotlinx-coroutines-core-1.3.5.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.3.72/8032138f12c0180bc4e51fe139d4c52b46db6109/kotlin-stdlib-1.3.72.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.3.72/6ca8bee3d88957eaaaef077c41c908c9940492d8/kotlin-stdlib-common-1.3.72.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.9.6.RELEASE/b98b670a8dc1079b17ff5dd82cc78acac001999a/reactor-netty-0.9.6.RELEASE.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.4.RELEASE/af99eff1cc01ccae33069e21c1db45328389747e/reactor-core-3.3.4.RELEASE.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.28/28f6099100b6309e93953d73e73470f2ce7e4c3a/classgraph-4.8.28.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.testcontainers/jdbc/1.14.1/b617d34b04a7d4a11e546d0174046986e6f9ab53/jdbc-1.14.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.ongres.scram/client/2.1/4ab4f013dce4ce2bdf315e3cfef529bdc2b6a5d1/client-2.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.testcontainers/database-commons/1.14.1/950abc75cddfa565ee810a136d2620f48bde86f/database-commons-1.14.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.testcontainers/testcontainers/1.14.1/defd04ff6ffc93e1ff988024048e8ba5bd298df3/testcontainers-1.14.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/17.0.0/8ceead41f4e71821919dbdb7a9847608f1a938cb/annotations-17.0.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.ongres.scram/common/2.1/43dfdba15afc165a2a5da75f3ef1a2f5f7300f61/common-2.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.48.Final/f7a0e40d0eea9a60c05f705a128473f08a3de1dc/netty-codec-http2-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.48.Final/79910203db6a542189c9aaa9ec187bf5ee16248a/netty-handler-proxy-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.48.Final/8b18499a51367d291efd1f200a4e865370d38738/netty-codec-http-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.48.Final/d459919c87dc1a2bf400b39053761ecc11c2e436/netty-handler-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.48.Final/2b25233dda8b986297076a7d62552c1ec2f41ed8/netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.ongres.stringprep/saslprep/1.1/eaf05b6446126f95de2f183b91751a854c2fc0bc/saslprep-1.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.48.Final/42b55c9ca8672ebb7160cadf3c70ffc11638f021/netty-codec-socks-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.48.Final/3142078325d745228da9d6d1f6f9931c63aaba16/netty-codec-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.48.Final/fb0a76d5d201b59ec2b3721c66eda737b3176f30/netty-transport-native-unix-common-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.48.Final/d285e8d9af5be1fc146b91a1b0d211e7a6a7e7d4/netty-transport-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.48.Final/7fd3ebb659c628e0158c9c2c971f4d0734d7f4f0/netty-buffer-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.48.Final/612ad2acc848ce46496095683eea621094763b80/netty-resolver-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.48.Final/ec27d04fd5e7b65ac171cd64734f9e4f1fc2a286/netty-common-4.1.48.Final.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.rnorth/tcp-unix-socket-proxy/1.0.2/cf53989130986c60113032e25185f4496ffbc186/tcp-unix-socket-proxy-1.0.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.30/b5a4b6d16ab13e34a88fae84c35cd5d68cac922c/slf4j-api-1.7.30.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.20/b8df472b31e1f17c232d2ad78ceb1c84e00c641b/commons-compress-1.20.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.rnorth.duct-tape/duct-tape/1.0.8/92edc22a9ab2f3e17c9bf700aaee377d50e8b530/duct-tape-1.0.8.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.rnorth.visible-assertions/visible-assertions/2.1.2/20d31a578030ec8e941888537267d3123c2ad1c1/visible-assertions-2.1.2.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna-platform/5.5.0/af38e7c4d0fc73c23ecd785443705bfdee5b90bf/jna-platform-5.5.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.ongres.stringprep/stringprep/1.1/9149e6d6f97be4c2b641c89ea569cb4fde872972/stringprep-1.1.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/1.3/42a25dc3219429f0e5d060061f71acb49bf010a0/hamcrest-core-1.3.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.kohlschutter.junixsocket/junixsocket-native-common/2.0.4/726bd66a934dea39c817382986496fa4eda96411/junixsocket-native-common-2.0.4.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/com.kohlschutter.junixsocket/junixsocket-common/2.0.4/b4d1870bf903412533e0b79c6fcd402defcfc05b/junixsocket-common-2.0.4.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/5.5.0/e0845217c4907822403912ad6828d8e0b256208/jna-5.5.0.jar, /Users/christoph/.gradle/caches/modules-2/files-2.1/org.scijava/native-lib-loader/2.0.2/1451fa03954c5e31a358b411147de472b4dab92c/native-lib-loader-2.0.2.jar], mutators=[], features=[], dependencyAnalysisMaxDistance=-1, jvmArgs=[-Djava.awt.headless=true, -Xmx512m], numberOfThreads=12, timeoutFactor=1.25, timeoutConstant=4000, targetTests=[^r2dbcfun\..*Test$, ^r2dbcfun\..*.*\..*Test$], loggingClasses=[kotlin.jvm.internal, kotlin.Result], maxMutationsPerClass=0, verbose=true, failWhenNoMutations=true, outputs=[XML, HTML], groupConfig=TestGroupConfig [excludedGroups=[], includedGroups=[]], fullMutationMatrix=false, mutationUnitSize=0, shouldCreateTimestampedReports=true, detectInlinedCode=false, exportLineCoverage=false, mutationThreshold=0, coverageThreshold=0, mutationEngine=gregor, javaExecutable=null, includeLaunchClasspath=false, properties={}, maxSurvivors=-1, excludedRunners=[], includedTestMethods=[], testPlugin=junit5, useClasspathJar=false, skipFailingTests=false]
16:43:21 PIT >> FINE : System class path is /Users/christoph/.gradle/caches/modules-2/files-2.1/org.pitest/pitest-junit5-plugin/0.12/6e1ee587a3457b0e7d636a5ce2e554fac62ec99/pitest-junit5-plugin-0.12.jar:/Users/christoph/.gradle/caches/modules-2/files-2.1/org.pitest/pitest-command-line/1.5.1/6f9357d32f5ab433f4f0ec668fcdf04de744d360/pitest-command-line-1.5.1.jar:/Users/christoph/.gradle/caches/modules-2/files-2.1/org.pitest/pitest-entry/1.5.1/fafe3010bf8d03c330ecd800103fc081b507aa4d/pitest-entry-1.5.1.jar:/Users/christoph/.gradle/caches/modules-2/files-2.1/org.pitest/pitest/1.5.1/3914ec0629c931560e0124247c9221aa0cd1c6d/pitest-1.5.1.jar
16:43:21 PIT >> FINE : Maximum available memory is 8192 mb
Exception in thread "main" org.pitest.help.PitHelpError: The class r2dbcfun.R2dbcRepo$findBy$$inlined$map$1$2$1 does not contain source debug information. All classes must have an associated source file
See http://pitest.org for more details.
        at org.pitest.mutationtest.verify.DefaultBuildVerifier.lambda$throwErrorIfHasNoSourceFile$2(DefaultBuildVerifier.java:68)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
        at org.pitest.functional.FCollection.forEach(FCollection.java:34)
        at org.pitest.mutationtest.verify.DefaultBuildVerifier.verify(DefaultBuildVerifier.java:36)
        at org.pitest.mutationtest.tooling.MutationCoverage.verifyBuildSuitableForMutationTesting(MutationCoverage.java:223)
        at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:111)
        at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:121)
        at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:51)
        at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:87)
        at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)

FAILURE: Build failed with an exception.

Its totally possible that this is a bug in the kotlin compiler, where it does not mark methods as synthetic, but I'm posting it here first to get more feedback, and maybe create a kotlin ticket later when I know more.

this is with pitest 1.5.1 and kotlin 1.3.72

probably related to #260

christophsturm commented 4 years ago

this is how the class looks in "show kotlin bytecode": https://gist.github.com/christophsturm/1f46f14dde1a61a14ce449b0632f883c

hcoles commented 4 years ago

Interestingly the source file name that pitest needs looks to be in there in a kotlin specific annotation, so we could start to read that when the normal debug info is not present.

Don't know enough about kotlin or co-routines to have an idea if it would make sense to mutate any of the code in this class though.

christophsturm commented 4 years ago

right now i can't use pitest at all, so just reading the annotation would be a huge improvement even if we don't mutate anything. the coroutines code is just inlined library code so i probably no need to mutate it.

christophsturm commented 4 years ago

btw, if pitest needs a line number for every non synthetic method, I wonder how it works with kotlin data classes, those have methods like toString and equals, which are not synthetic and are autogenerated.

timothyolt commented 2 years ago

Just an FYI for anyone else looking at this, you can use this to exclude these from mutation testing. Not ideal, but it gets pitest working on the rest of the source.

pitest {
    excludedClasses = ['*$special$$inlined$*']
}
hcoles commented 6 months ago

Kotlin support is now available in https://docs.arcmutate.com/docs/kotlin