runtime-only Dependencies not added to Pitest classpath

[Pfoerd]

Describe the bug Dependencies that are configured to be "runtime only" are not added to Pitest classpath. E.g. when using mockk 1.11.0 it ships transitive dependencies mockk-agent-jvm and kotlin-reflect with scope "runtime" (see Maven Central) but Pitest throws

8:50:45 AM PIT >> INFO : MINION : java.lang.NoClassDefFoundError: kotlin/reflect/full/KClasses
8:50:45 AM PIT >> INFO : MINION :       at io.mockk.impl.JvmMockKGateway.<init>(JvmMockKGateway.kt:185)
8:50:45 AM PIT >> INFO : MINION :       at io.mockk.impl.JvmMockKGateway.<clinit>(JvmMockKGateway.kt:173)

Verbose output shows that the mockk-agent-jvm and kotlin-reflect are not part of classPathElements passed to Pitest.

To Reproduce Steps to reproduce the behavior:

1. Standard project setup
2. Add testImplementation("io.mockk:mockk:1.11.0") dependency
3. Use mockk in one of your tests
4. Test is "green" when executed as unit test but Pitest fails:
   • Fails with "Exception in thread "main" org.pitest.help.PitHelpError: xxx tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite."
   • shows something like java.lang.NoClassDefFoundError: Could not initialize class io.mockk.impl.JvmMockKGateway in verbose output of Pitest

Expected behavior Pitest successfully finishes without exceptions

Version used: pl.droidsonroids.pitest 0.2.9

Known Workaround Discover the dependency that is missing via Pitest verbose output and add it as a "compile" dependency to your project

Suggested Solution Add all entries of *UnitTestRuntimeClasspath* configurations (or something like that) to additionalClasspath

[Pfoerd]

Hi @koral-- any comments on this?

[koral--]

Sorry, not yet.

[koral--]

@Pfoerd Could you create and share a simple reproducer project? Or better create or modify the functional test with mockk at https://github.com/koral--/gradle-pitest-plugin/tree/master/src/funcTest/resources/testProjects/mockk?

I was unable to reproduce your issue on plugin version 0.2.12.

[Pfoerd]

@koral-- I created a branch that showcases the problem. I found out that the problem only occurs in older versions of Mockk prior to 12.x because Mockk fixed their recommended project setup.

With Mockk 11.0.0 (and lower) you will run into the issue I described here if you follow the suggested setup for those releases (without explicitly declaring a testImplementation dependency to the agent-jvm library). The reason is a missing class:

Caused by: java.lang.ClassNotFoundException: io.mockk.proxy.MockKAgentFactory

Root cause is that the transitiv dependency "io.mockk:mockk-agent-jvm:1.11.0" seems to be just a "runtimeOnly" dependency of mockk (see Maven Central "Runtime dependencies") and is therefore currently not included in Pitest's classpath.

[koral--]

OK, so if it is already fixed issue on mockk side, it is not worth adding any workaround in the pitest plugin code. If you can't/don't want to update mockk, you can add an explicit runtimeOnly dependencies to your project's buildscript:

    testRuntimeOnly "io.mockk:mockk-agent-jvm:1.11.0"
    testRuntimeOnly "org.jetbrains.kotlin:kotlin-reflect:1.8.21" //not sure about the minimum version here

[Pfoerd]

Thank you for having a look at this. My issue here isn't just the specific issue with Mockk but also the general problem that testRuntimeOnly dependencies are never added to the pitest classpath. Imho this may probably cause other issues even in other scenarios where no mockk is involved!?

[koral--]

From what I can see there is a difference in POMs of mockk. Since version 1.12.8 the following lines were added:

    <dependency>
      <groupId>io.mockk</groupId>
      <artifactId>mockk-agent</artifactId>
      <version>1.12.8</version>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>io.mockk</groupId>
      <artifactId>mockk-agent-api</artifactId>
      <version>1.12.8</version>
      <scope>runtime</scope>
    </dependency>

https://central.sonatype.com/artifact/io.mockk/mockk/1.12.8 There were neither mockk-agent nor mockk-agent-api in 1.12.7 or older.

I don't see how it can be done from the plugin side automatically (except for hardcoding mockk artifact names which doesn't look good). Do you have any idea how to do that? Or maybe do you have another example of missing runtime dependencies?

[Pfoerd]

Mhh looking a bit deeper into this I have to say I don't really understand whats going on. Maybe it has to do something with this issue.

Since 1.12.1 Mockk recommends the following setup:

testImplementation "io.mockk:mockk:{version}"
testImplementation "io.mockk:mockk-agent-jvm:{version}"

so all dependencies are there through direct dependency to mockk-agent-jvm (mockk-agent-jvm 1.12.1. has dependencies to mockk-agent-api and mockk-agent-common

In 1.12.0 (or older) the recommendet setup was just:

testImplementation "io.mockk:mockk:{version}"

but mockk 1.12.0 (or older) itself has (transitive) dependencies to all required libraries (mockk-agent-jvm 1.12.0. has dependencies to mockk-agent-api and mockk-agent-common)

<dependency>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-stdlib</artifactId>
  <version>1.3.72</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>io.mockk</groupId>
  <artifactId>mockk-agent-jvm</artifactId>
  <version>1.12.0</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-reflect</artifactId>
  <version>1.3.72</version>
  <scope>runtime</scope>
</dependency>

So I would say: even in mockk 1.11.0 etc. all required dependencies should be defined correctly or am I missing something?

[koral--]

According to this issue https://github.com/mockk/mockk/issues/605 which caused the recommended mockk setup change and which is also linked to SO thread you mentioned, it seems that such behavior is caused by Android Gradle plugin.

The trick is in Android Gradle Plugin. It builds (at least) 2 classpathes: compile and runtime. IDE Android Plugin (in Android Studio) imports only libraries from Compile classpath (runtime entries are removed). When running tests with the Gradle runner, IDE delegates execution to gradle, and Gradle invokes tests with the Runtime classpath. When running tests with JUnit in Android Studio, JUnit uses runtime classpath of an IDE module as it's seen by the IDE (which already has no runtime libs because they were removed during import).

Source: https://stackoverflow.com/a/68990942/630398

kotlin-reflect runtime dependency also seems to be removed due to the same reason.

So it seems that the only thing I can do from the plugin side is to add some note to a readme about that.

[Pfoerd]

As i just said I'm not a gradle expert so maybe my questions are a bit naiv, sorry for that. If this statement holds: The trick is in Android Gradle Plugin. It builds (at least) 2 classpathes: compile and runtime. then maybe the pitest-gradle-plugin could be able to pass both the compile and runtime classpath to pitest?

[koral--]

Good point, I'll check if they are both available via its API.

[Pfoerd]

@koral-- Did you have a chance to look at this?

[koral--]

I've looked a little bit but haven't found the solution yet. I'm going to reserve more time for this soon.

[koral--]

It took a long time, but it seems that I finally managed to fix this issue.