Exception NoClassDefFoundError org/testcontainers/utility/PathUtils

[nicolas-lattuada-n26]

Hello

There is an exception in MountableFile, it does not find the class PathUtils when it calls recursiveDeleteDir from shutdownHook. I think this is because during the shutdown hook the class loader is already closed by Maven and cannot be used to load new classes.

See attached stack trace

Exception in thread "Thread-17" java.lang.NoClassDefFoundError: org/testcontainers/utility/PathUtils$1
    at org.testcontainers.utility.PathUtils.recursiveDeleteDir(PathUtils.java:26)
    at org.testcontainers.utility.MountableFile.lambda$deleteOnExit$0(MountableFile.java:284)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: org.testcontainers.utility.PathUtils$1
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
    ... 3 more

[bsideup]

@nicolas-lattuada-n26 could you please create an example to reproduce this?

[nicolas-lattuada-n26]

Unfortunately this is included in proprietary code that I cannot disclose here, but I think if you include a call to MountableFile from a maven plugin you should be able to reproduce it.

[bsideup]

@nicolas-lattuada-n26 sorry, but this sounds too specific. It may also be a bug in Maven's classloading mechanism, so I suggest creating a minimal reproducer to verify it.

[nicolas-lattuada-n26]

Hello @bsideup Thanks for your reply, it took me some time to create a minimum project so that you can reproduce. https://github.com/nicolas-lattuada-n26/sample-plugin cc/ @Osguima3

[nicolas-lattuada-n26]

Hello @bsideup Have you been able to run the test project provided yet?

[flylo]

Running into this same issue

[DuckyCh]

Same error here.

[rnorth]

Hmm, I'm not really sure if we can help much with this. Running Testcontainers code directly from a Maven plugin isn't something we've attempted to support, as we're more focused on testing. I assume your usage scenarios are not testing?

As you say it does look like the Maven classloader closing is what's causing it. If that's Maven's classloader behaviour then I imagine shutdown hooks are essentially not safe inside of a Maven plugin, and should be avoided.

If there's something simple that we could change then I think we could accept a PR, but otherwise I'm afraid we might have to chalk this up as an unsupported use case. Sorry to disappoint.

[heruan]

I'm still suffering this. My Maven plugin is for testing purposes and runs a few database containers, which per se succeeds but I get that exception at last which fails the CI build.

[flylo]

@heruan fwiw I just run this stupid util in a finally block to suppress the exceptions. Could have some bad side-effects, however.

package com.lol.utils;

import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.Map;

public class PwnageUtils {

  // removes the shutdown hooks from testcontainers threads because they will fail after mojo closure context closes.
  // this is obviously horrible, but it stops us from getting pwned by classloader exceptions after successful builds.
  public static void stopPwnage() {
    try {
      Class clazz = Class.forName("java.lang.ApplicationShutdownHooks");
      Field field = clazz.getDeclaredField("hooks");
      field.setAccessible(true);
      Map<Thread, Thread> hooks = (Map<Thread, Thread>) field.get(null);
      // Need to create a new map or else we'll get CMEs
      Map<Thread, Thread> hookMap = new HashMap<>();
      hooks.forEach(hookMap::put);
      hookMap.forEach((thread, hook) -> {
        if (thread.getThreadGroup().getName().toLowerCase().contains("testcontainers")) {
          Runtime.getRuntime().removeShutdownHook(hook);
        }
      });
    } catch (Exception e) {
      throw new RuntimeException("pwned", e);
    }
  }
}

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[stale[bot]]

This issue has been automatically closed due to inactivity. We apologise if this is still an active problem for you, and would ask you to re-open the issue if this is the case.

[auriium]

This is still an active issue for me. I'm using testcontainers in a maven plugin in order to coordinate code generation with flyway and jooq, and believe this would still be useful. A solution I propose would be adding an API method to allow starting of the testcontainer without shutdown hooks

[bsideup]

@Auriium shutdown hooks are an essential part of JVM, and NoClassDefFoundError is caused by Maven, not Testcontainers.

However, you can call ResourceReaper.instance().stopAndRemoveContainer(...) after you finished with it. This way, it will be removed from the set of containers scheduled for removal, and the shutdown hook won't load any classes.

[bsideup]

@Auriium there is also a potential for contribution - empty registeredContainers and other collections in ResourceReaper#performCleanup, so that you can call it after you perform the task.

[auriium]

@bsideup okay, sounds good: Can i ask, if removing the container via the resource reaper removes it from the set of containers scheduled for removal and fixes the issue with the shutdown hook, why doesn't GenericContainer#stop (or whatever it's called) or alternatively the close method from the autocloseable interface manually invoke this stopAndRemoveContainer to not cause the issue in the first place?

[bsideup]

@Auriium stop does call stopAndRemoveContainer: https://github.com/testcontainers/testcontainers-java/blob/57ce0c4861d83c804fc1eafb0d2239f35726553f/core/src/main/java/org/testcontainers/containers/GenericContainer.java#L601

Also, I just realized that the error comes from another hook registered in MountableFile. The tricky thing here is that Java does not support the deletion of non-empty folders, so we have to call custom code to recursively perform the deletion.

Have you tried reporting the issue to Maven, btw?

[auriium]

Okay, if stop does call stopAndRemoveContainer then how does calling manually stopAndRemoveContainer fix the issue?

[auriium]

Oh just read your new response, is the mountablefile hook the root cause of the exception, even if the resourcereaper method is called?

[simasch]

Like @auriium I'm also using Testcontainers to generate jOOQ code. The testcontainer is started with groovy-maven-plugin.

With PostgreSQLContainer I don't see this exception but with MariaDBContainer I get

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36.006 s
[INFO] Finished at: 2022-01-21T13:16:09+01:00
[INFO] ------------------------------------------------------------------------
Exception in thread "Thread-18" java.lang.NoClassDefFoundError: org/testcontainers/utility/PathUtils
    at org.testcontainers.utility.MountableFile.lambda$deleteOnExit$0(MountableFile.java:296)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: org.testcontainers.utility.PathUtils
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
    ... 2 more

[kiview]

@simasch The original explanation by @bsideup is still valid: https://github.com/testcontainers/testcontainers-java/issues/1454#issuecomment-801699021

Also, see the discussion in this closed PR: https://github.com/testcontainers/testcontainers-java/pull/1746#issuecomment-541342639

[simasch]

Hi @kiview I've read both discussions and think I understand. What I don't understand that the error only happens with MariaDBContainer.

[kiview]

Do you have exactly the same code to interact with both containers? I looked into our implementation of MariaDBContainer and could not find any usage of MountableFile.

[simasch]

Yes

db = new org.testcontainers.containers.PostgreSQLContainer("postgres:12.7")
                                .withUsername("${db.username}")
                                .withDatabaseName("jtaf4")
                                .withPassword("${db.password}")
                                db.start()
                                project.properties.setProperty('db.url', db.getJdbcUrl())

[jjijmker]

I ran into the same issue using an Oracle image (GenericContainer) inside a Maven plugin, and managed to avoid the use of MountableFile (and with that the issue with the hook) by using Transferable rather than MountableFile to create the required files on the running container:

oracleContainer.copyFileToContainer(
    Transferable.of(recreateSchemaSql),
    getSchemaRecreateScriptContainerPath());

This might not be an option for everyone but it helped me.

[uldall]

Did anyone ever find a solution to this issue? I also get the error with jOOQ+flyway.

Note, I only seem to get it when TESTCONTAINERS_RYUK_DISABLED=true, which is required on Bitbucket Pipelines.

[McPringle]

As anybody can see in the comments here, this issue is not fixed and should be reopened. It was closed by some clean-up script because it was stale.

I use the latest version of testcontainers and can reproduce this issue on every Maven run:

src/test/application.properties:

spring.datasource.url=jdbc:tc:mariadb:10.11.2:///test?allowMultiQueries=true
spring.datasource.username=test
spring.datasource.password=test

pom.xml:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>1.19.3</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mariadb</artifactId>
    <version>1.19.3</version>
</dependency>

Output of './mvnw verify':

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30.406 s
[INFO] Finished at: 2023-12-06T09:42:59+01:00
[INFO] ------------------------------------------------------------------------
Exception in thread "Thread-3" java.lang.NoClassDefFoundError: org/testcontainers/utility/PathUtils
    at org.testcontainers.utility.MountableFile.lambda$deleteOnExit$0(MountableFile.java:318)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: org.testcontainers.utility.PathUtils
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
    ... 2 more

Steps to reproduce:

$ git clone git@github.com:komunumo/komunumo-server.git
$ cd komunumo-server
$ ./mvnw verify

I'm happy to provide more information if needed.

[bedla]

I am getting same issue with different class. Problem is, as mentioned here, with closed classloader/classworld-realm after maven-plugin finished it's work.

I dug little bit deeper and found that seems impossible because it is how Maven manages classloaders.

After it finishes work, it dispose all resources (which is good practice), but we already have registered ShutdownHooks to classes loaded but maven-plugin classloaders/realms.

See line 299 https://github.com/apache/maven/blob/maven-3.9.5/maven-embedder/src/main/java/org/apache/maven/cli/MavenCli.java#L268-L302

IMHO this cannot be fixed easily, fix might be to use classes from main classloader (JDK) only.

For the record (and other to find this with google), my exception looks like this:

Exception in thread "Thread-28" java.lang.NoClassDefFoundError: org/testcontainers/shaded/com/github/dockerjava/core/exec/KillContainerCmdExec
    at org.testcontainers.shaded.com.github.dockerjava.core.AbstractDockerCmdExecFactory.createKillContainerCmdExec(AbstractDockerCmdExecFactory.java:366)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientImpl.killContainerCmd(DockerClientImpl.java:473)
    at com.github.dockerjava.api.DockerClientDelegate.killContainerCmd(DockerClientDelegate.java:280)
    at com.github.dockerjava.api.DockerClientDelegate.killContainerCmd(DockerClientDelegate.java:280)
    at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$0(RyukResourceReaper.java:86)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: org.testcontainers.shaded.com.github.dockerjava.core.exec.KillContainerCmdExec
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
    ... 6 more

[oxygenecore]

So, the ticket is closed, and what is the resolution for the users of testcontainers library? To not use it?

I'm using testcontainers JDBC URL, and just provide it to the two plugins I don't build myself, so can't really call any java api. So, for my case there is no solution?

[oxygenecore]

Hmm, I'm not really sure if we can help much with this. Running Testcontainers code directly from a Maven plugin isn't something we've attempted to support, as we're more focused on testing. I assume your usage scenarios are not testing?

As you say it does look like the Maven classloader closing is what's causing it. If that's Maven's classloader behaviour then I imagine shutdown hooks are essentially not safe inside of a Maven plugin, and should be avoided.

If there's something simple that we could change then I think we could accept a PR, but otherwise I'm afraid we might have to chalk this up as an unsupported use case. Sorry to disappoint.

Maybe I don't understand something, but why then have this plugin in the first place? Plugin repo (also testcontainers) - https://github.com/testcontainers/testcontainers-jooq-codegen-maven-plugin The official how-to page also refers it: https://testcontainers.com/guides/working-with-jooq-flyway-using-testcontainers/

Looks like a big inconsistency to me.

[oxygenecore]

This is what Maven documentation states: https://maven.apache.org/plugin-developers/common-bugs.html#using-shutdown-hooks

The problem is that the JVM executing Maven can be running much longer than the actual Maven build. Of course, this does not apply to the standalone invocation of Maven from the command line. However, it affects the embedded usage of Maven in IDEs or CI servers. In those cases, the cleanup tasks will be deferred, too. If the JVM is then executing a bunch of other Maven builds, many such cleanup tasks can sum up, eating up resources of the JVM.

For this reason, plugin developers should avoid usage of shutdown hooks and rather use try/finally blocks to perform cleanup as soon as the resources are no longer needed.

So, I guess, the maven plugin should clean up the shutdown hooks created by testcontainers-java?

[kiview]

I just re-opened the issue, since it is clearly amplified by https://github.com/testcontainers/testcontainers-jooq-codegen-maven-plugin,