search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.78k stars 2.68k forks source link

ClassLoader Leak using TestProfiles (OutOfMemoryError: Metaspace) #38774

Open xuckz opened 8 months ago

xuckz commented 8 months ago

Describe the bug

Using multiple TestProfiles results in OutOfMemoryError: Metaspace It seems that running a @QuarkusTest with its own @\TestProfile is not cleaned up properly after the Test has finished, and all QuarkusClassLoaders used are permanently stored in MetaSpace.

Each Test adding its own ClassLoader to MetaSpace until the OutOfMemoryError occurs.

Increasing the MetaSpace for each @QuarkusTest with its own @\TestProfile is a limited workaround.

Edit: I think this might be related to https://github.com/quarkusio/quarkus/issues/12498 since the Heap Space is also effected.

Edit2: Screenshot added

image-2024-02-13-13-01-49-831

Expected behavior

The expected behavior would be, that you are not limited to any amount of @\TestProfil and that each @\TestProfile would not increase the amount of MetaSpace required.

Actual behavior

Each @QuarkusTest with its own @\TestProfile requires the MetaSpace increased.

How to Reproduce?

Ready to go repo: https://github.com/xuckz/quarkus-quickstarts-classloader-leak-demo https://github.com/xuckz/quarkus-quickstarts-classloader-leak-demo/commit/ccfa14470a8e540336ca086768546cb7cdfe53f1

I used the quarkus-quickstarts/hibernate-orm-panache-quickstart as showcase:

<plugin>
    <!-- you need this specific version to integrate with the other build helpers -->
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire-plugin.version}</version>
    <configuration>
        <argLine>-XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256M</argLine>
        <systemPropertyVariables>
            <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
            <maven.home>${maven.home}</maven.home>
        </systemPropertyVariables>
    </configuration>
</plugin>
package org.acme.hibernate.orm.panache.test;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.QuarkusTestProfile;
import io.quarkus.test.junit.TestProfile;

@QuarkusTest
@TestProfile(LeakHunt1Test.LeakHuntTestProfile.class)
public class LeakHunt1Test {

    public static class LeakHuntTestProfile implements QuarkusTestProfile {
    }

    @Test
    void test() {
        System.out.println("asdfasdf");
    }
}

Output of uname -a or ver

Linux xxx 5.4.0-167-generic #184-Ubuntu SMP Tue Oct 31 09:21:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "17.0.9" 2023-10-17 OpenJDK Runtime Environment (build 17.0.9+9-Ubuntu-120.04) OpenJDK 64-Bit Server VM (build 17.0.9+9-Ubuntu-120.04, mixed mode, sharing)

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.6.3 Maven home: /usr/share/maven Java version: 17.0.9, vendor: Private Build, runtime: /usr/lib/jvm/java-17-openjdk-amd64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.4.0-167-generic", arch: "amd64", family: "unix"

Additional information

No response

quarkus-bot[bot] commented 8 months ago

/cc @geoand (testing)

Adelrisk commented 7 months ago

I don't want to be a "I'm having this problem too" commentator, but ... I'm having this problem too. But with only three profiles, each applied to a different test case/test class.

Just for an idea of the scope: real-world project, kotlin and gradle, two entities, few endpoints, 23 passing tests and good coverage (this service is very CRUD-like). (A useful utility microservice that does a single, small job well.)

I then introduced three test profiles that configure two properties differently, such as:

class NoSsiIssuanceV1Profile: QuarkusTestProfile {
    override fun getConfigOverrides(): Map<String, String> {
        return mapOf(
            "quarkus.rest-client.ssi-issuance-v1-api.url" to "",
            "quarkus.rest-client.ssi-issuance-v2-api.url" to "http://ssi-v1:8080"
        )
    }
}

I use each profile once to verify the behaviour of injection I programmed into the application, such as:

@QuarkusTest
@TestProfile(NoSsiIssuanceProfile::class)
open class NoSsiClientTest {

    @Inject
    lateinit var sut: Set<ApiVersion>
    @Inject
    lateinit var v1: Optional<SsiIssuanceV1HttpClient>
    @Inject
    lateinit var v2: Optional<SsiIssuanceV2HttpClient>

    @Test
    fun hasCorrectApiVersions() {
        assertThat(sut, equalTo(setOf()))
    }
    @Test
    fun hasNoV1Client() {
        assertThat(v1.isEmpty, equalTo(true))
    }
    @Test
    fun hasNoV2Client() {
        assertThat(v2.isEmpty, equalTo(true))
    }
}

When I run all my tests, the JVM prematurely aborts the execution of all three test classes marked with @TestProfile. As an experiment, I set a filter so only 10 other tests were executed, in which case "only" one test class with a test profile aborted due to OOM.

This gives the impression that the complexity threshold is quite low before this bug leads to an OutOfMemoryError.

geoand commented 7 months ago

Can you attach your sample application that leads to the OOME?

Guillaume-Lebegue commented 7 months ago

I also have that problem on my project We use around 10 different profiles using multiple databases, running Kafka or a scheduler.
And today when I added a new one my CI crashed with an OOME.

After checking, I noticed that each new Profile created a jump in memory use.
And by the last test, the JVM was using 6G of RAM, which is my runner's limit, so OOME.

I made a quick repo to show the problem https://github.com/Guillaume-Lebegue/quarkus_test_out_of_memory_testprofile
In it, 10 tests, each with its profile, and each profile and test are doing the same thing.

When running the tests, the first test ended with ~500Mo of memory used, and the last test was at 1200Mo of memory used.
I also created another branch only_one_profile that makes all 10 tests run on the same profile; this time, it did not go more than 500Mo.

janek64 commented 6 months ago

I am facing the same issue with only two test profiles. All tests of the first profile are running as expected, but the startup of the second profile fails with an OutOfMemoryError.

A new observation that I can add is that the issue only occurs since I added io.quarkus:quarkus-jacoco to my project. When I remove the dependency, all tests are running as expected.

billapepper commented 5 months ago

Another "me too", but I recently started migrating from 3.2.x to 3.8.4 (as the LTS is expiring on 3.2.x in July).

I ran into a problem where @InjectSpy began throwing a "Please don't pass mock here. Spy is not allowed on mock." error when trying to spy using a smallrye config... this seemingly stems from the Mockito upgrade (https://github.com/mockito/mockito/pull/2989/files)... but I couldn't find a workaround, so instead, I reworked my application to use @TestProfile and override the configuration for the specific tests. Very basic profiles, in a similar fashion to how https://github.com/quarkusio/quarkus/issues/38774#issuecomment-1999958050 did:

    public static class PublicPrivateDomains implements QuarkusTestProfile {

        @Override
        public Map<String, String> getConfigOverrides() {
            return Map.of(
                    "prime.landing-service.public-domains", ".domain.com",
                    "prime.landing-service.private-domains", ".domain.net"
            );
        }
    }

However, since doing this, the build with several of these tests fails with a OOM error. I have tried increasing the memory using <argLine>-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=2048M -XX:MaxRAMPercentage=75</argLine> on the maven-surefire-plugin, but it still runs out of memory in our GH actions.

These tests ran without issues when they were not using separate @TestProfiles.

Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
Error:  Metaspace
Error:      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:628)
Error:      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
Error:      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
Error:      at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
Error:      at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
Error:      at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
Error:      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
Error:      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
Error:      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
Error:      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
Error:      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:***1)
Error:      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
Error:      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:1***)
Error:      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
Error:      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
Error:      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
Error:      at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:299)
Error:      at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
Error:      at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
Error:      at org.apache.maven.cli.MavenCli.execute(MavenCli.java:963)
Error:      at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:296)
Error:      at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
Error:      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Error:      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Error:      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
Error:      at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
Error:      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
Error:      at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

EDIT: Also, removing quarkus-jacoco did not seem to stop the problem for me (with regards to https://github.com/quarkusio/quarkus/issues/38774#issuecomment-2055674635)

aylwyne commented 5 months ago

I'm wondering if there is any traction on this issue. This is blocking our ability to upgrade to 3.8.4 because our test suite uses test profiles quite a lot and we haven't been able to get any alternatives working (see previous comment from billapepper). We are currently on the previous LTS version (3.2.x) which is expiring soon yet are blocked on upgrading to the 3.8.x LTS release.

geoand commented 4 months ago

41156 might be related and we have a couple of sample changes that may improve things.