search

Kotlin / dokka

API documentation engine for Kotlin
https://kotl.in/dokka
Apache License 2.0
3.36k stars 397 forks source link

dokkaJavadocJar fires "java.lang.OutOfMemoryError: Metaspace" #1405

Closed elect86 closed 1 year ago

elect86 commented 3 years ago

Question

I added the following in my build.gradle.kts

val dokkaJavadocJar by tasks.register<Jar>("dokkaJavadocJar") {
    dependsOn(tasks.dokkaJavadoc)
    from(tasks.dokkaJavadoc.get().outputDirectory.get())
    archiveClassifier.set("javadoc")
}

val dokkaHtmlJar by tasks.register<Jar>("dokkaHtmlJar") {
    dependsOn(tasks.dokkaHtml)
    from(tasks.dokkaHtml.get().outputDirectory.get())
    archiveClassifier.set("html-doc")
}

val sourceJar = task("sourceJar", Jar::class) {
    dependsOn(tasks["classes"])
    archiveClassifier.set("sources")
    from(sourceSets.main.get().allSource)
}

artifacts {
    archives(dokkaJavadocJar)
    archives(dokkaHtmlJar)
    archives(sourceJar)
}

However dokkaJavadocJar produces the titled error:

java.lang.OutOfMemoryError: Metaspace

and the following

Task :compileJava FAILED

But I'm pretty sure is dokkaJavadocJar the error root because if I comment it out, it works flawless

In order to fix this, I added in gradle.properties

org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m

Installation

I wonder if this is "fine" or something nasty is happening in the background I shall be aware of

Ps: the very same problem happens also in other projects

Kordyjan commented 3 years ago

This is really bizarre. Dokka is indeed memory-intensive in some parts, which are not yet optimized, but it should only affect heap space. I don't know what could cause overflow of metaspace. We will definitely profile that.

Kordyjan commented 3 years ago

Also thanks for suggesting a workaround. I'll repost it for the better visibility.

Workaround:

put org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m in gradle.properties

Kordyjan commented 3 years ago

Could you check if this is reproducible if you run ./gradlew --stop before running any tasks or even put org.gradle.daemon=false in your gradle.properties?

gabrielittner commented 3 years ago

If you need a project to reproduce it happens in the integration tests of our Gradle plugin which runs dokka.

The commit https://github.com/vanniktech/gradle-maven-publish-plugin/commit/3ef0227c39a64336f8cdb1c65504e08646e7c2a6 The ci run https://travis-ci.org/github/vanniktech/gradle-maven-publish-plugin/builds/723577862 (also happens locally when running ./gradlew integrationTest)

elect86 commented 3 years ago

Could you check if this is reproducible if you run ./gradlew --stop before running any tasks or even put org.gradle.daemon=false in your gradle.properties?

Both seem working

elect86 commented 3 years ago

What does this have to do with the daemon, @Kordyjan?

elect86 commented 3 years ago

Just got again java.lang.OutOfMemoryError: Metaspace with org.gradle.daemon=false

altavir commented 3 years ago

I get the same error for dokkaHtml for a not-so-large multi-module project. And yes, it is reproducible. Enlarging metaspace seems to be working.

gciatto commented 3 years ago

I'm getting the same problem here, with or without the daemon. I'm working on a multi-platform & multi-project project. Here's the branch I'm working on: https://gitlab.com/pika-lab/tuprolog/2p-in-kotlin/-/commits/feature/dokka-1.4

joffrey-bion commented 3 years ago

In my case I had to bump the metaspace even further (1024m). I have a multi-module project with 8 modules, half of which are multiplatform. Note that I'm running dokkaHtml, not dokkaJavadoc, but I get the same error with javadoc.

ansman commented 3 years ago

I believe one reason for this is parallel execution which spawns multiple processes which fight over the same resource. Running with the --no-parallel seems to help, at least for our project.

ansman commented 3 years ago

Although --no-parallel gets further (as in more modules get completes), eventually I do hit the same error. This could indicate that this is caused by a memory leak.

512MB is not enough for my project, I also had to bump to 1024. I wouldn't be surprised you need even more if you have more modules

ansman commented 3 years ago

Any updates on this? It's turning out to be a pretty massive issue because we're having trouble publishing our library that has lots of modules. Happy to spend some time debugging if someone can point me in the right direction.

ansman commented 3 years ago

This issue is even more problematic since the Gradle Daemon doesn't die but rather hangs so you have to manually kill it. The error in question something like this:

  Process exited with code 1
    Gradle failure report
    FAILURE: Build failed with an exception.

    * What went wrong:
    Gradle could not start your build.
    > Could not create service of type ResourceSnapshotterCacheService using GradleUserHomeServices.createResourceSnapshotterCacheService().
       > Timeout waiting to lock file hash cache (/Users/user/.gradle/caches/6.6/fileHashes). It is currently in use by another Gradle instance.
         Owner PID: 71496
         Our PID: 71896
         Owner Operation:
         Our operation:
         Lock file: /Users/user/.gradle/caches/6.6/fileHashes/fileHashes.lock
cohenor commented 3 years ago

This issue is even more problematic since the Gradle Daemon doesn't die but rather hangs so you have to manually kill it. The error in question something like this:

  Process exited with code 1
    Gradle failure report
    FAILURE: Build failed with an exception.

    * What went wrong:
    Gradle could not start your build.
    > Could not create service of type ResourceSnapshotterCacheService using GradleUserHomeServices.createResourceSnapshotterCacheService().
       > Timeout waiting to lock file hash cache (/Users/user/.gradle/caches/6.6/fileHashes). It is currently in use by another Gradle instance.
         Owner PID: 71496
         Our PID: 71896
         Owner Operation:
         Our operation:
         Lock file: /Users/user/.gradle/caches/6.6/fileHashes/fileHashes.lock

I'm having the exact same experience in a rather small multi-module project. Already tried to increase metaspace and to disable parallelism but it didn't provide any deterministic results. Any updates on this? 🙏

willbuck commented 3 years ago

I am also experiencing the same issue in https://github.com/micronaut-projects/micronaut-kotlin/tree/issue-78

Running --no-parallel with org.gradle.daemon=false and org.gradle.jvmargs=-XX:MaxMetaspaceSize=2g does seem to solve the problem for a clean build (though I do still see these bizarre warnings, see below) but that isn't really a practical solve for our github actions to publish our documentation.

Definitely seems a memory leak is the culprit, please let us know how profiling that bit goes!

WARN: error in opening zip file: /home/will/.gradle/caches/modules-2/files-2.1/org.graalvm.nativeimage/svm-hosted-native-linux-amd64/20.2.0/55f876b0688181a47436128f0e06b9ab93e17a32/svm-hosted-native-linux-amd64-20.2.0.tar.gz
java.util.zip.ZipException: error in opening zip file
WARN: error in opening zip file: /home/will/.gradle/caches/modules-2/files-2.1/org.graalvm.truffle/truffle-nfi-native-darwin-amd64/20.2.0/d3ad1011cba788a2b36792ee2ea128b90d5a4f8e/truffle-nfi-native-darwin-amd64-20.2.0.tar.gz
java.util.zip.ZipException: error in opening zip file
asarkar commented 3 years ago

Frequent OOM error in a fairly small project.

> Task :okgrpc-cli:dokkaHtml
Dokka 1.4.* is an alpha project
Initializing plugins
Validity check
Creating documentation models

Exception in thread "DefaultDispatcher-worker-8" java.lang.OutOfMemoryError: Metaspace
Daemon will be stopped at the end of the build after running out of JVM memory
Exception in thread "DefaultDispatcher-worker-1" java.lang.OutOfMemoryError: Metaspace
Exception in thread "DefaultDispatcher-worker-3" java.lang.OutOfMemoryError: Metaspace

Since the usual first response from any OOS maintainer is "have you tried the latest version", I'm running v1.4.10.2.

@willbuck A Javadoc engine shouldn't need 2G RAM to generate a dozen or so HTML files (that's how many for me).

ileasile commented 3 years ago

Mee too, pretty small project: https://github.com/Kotlin/kotlin-jupyter/tree/notebook-api Getting the error only on CI, locally I had org.gradle.jvmargs=-Xmx2048m, and it worked well.

MarvinSchramm commented 3 years ago

I have the same issue, simple multimodule object. This prevents me from publishing the libraries with javadoc at the moment

The gradle properties helps locally, but my CI system is github actions, i dont have 2G available there

ansman commented 3 years ago

Today I saw some new errors related to this:

...
java.lang.OutOfMemoryError: Metaspace
WARN: Could not read file: /Users/nicklas/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.4.20/9be77b243a362b745e365f286627b8724337009c/kotlin-stdlib-1.4.20.jar!/kotlin/SuspendKt.class; size in bytes: 1108; file type: CLASS
java.lang.OutOfMemoryError: Metaspace
WARN: Could not read file: /Users/nicklas/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.4.20/9be77b243a362b745e365f286627b8724337009c/kotlin-stdlib-1.4.20.jar!/kotlin/ThrowsKt.class; size in bytes: 540; file type: CLASS
java.lang.OutOfMemoryError: Metaspace
WARN: Could not read file: /Users/nicklas/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.4.20/9be77b243a362b745e365f286627b8724337009c/kotlin-stdlib-1.4.20.jar!/kotlin/TuplesKt.class; size in bytes: 1904; file type: CLASS
java.lang.OutOfMemoryError: Metaspace

This time it actually shows some more info on when you get OOM. It seems to be while reading the metadata/class files for the kotlin stdlib in my case.

This happened while generating gfm docs specifically.

andreihh commented 3 years ago

I'm getting the same issue on Travis CI for this project. This build works fine locally, but fails on CI server.

I tried everything suggested here (disabling the daemon, disabling parallel builds, increasing metaspace memory to 2 GB), but nothing worked.

marcphilipp commented 3 years ago

We're seeing the same issue with https://github.com/gradle-nexus/publish-plugin/.

willbuck commented 3 years ago

Anything any of us out in the user community can do to help resolve this issue? I'd really love to highlight dokka docs for https://github.com/micronaut-projects/micronaut-kotlin/tree/master/kotlin-extension-functions but this is a major thorn in doing so. I will read up on Contributing and see if I can step debug enough to figure out what I'm looking at but I'm afraid debugging gradle tasks is not my area of expertise.

asarkar commented 3 years ago

OSS maintainers often feel it's a thankless and underappreciated job, but any paid project with an OOM error for 6 months would be shut down and the project members would be looking for new jobs. But not if it's OSS :)

kamildoleglo commented 3 years ago

Long story short: it's a memory leak (mostly) caused by kotlinx.coroutines https://github.com/Kotlin/kotlinx.coroutines/issues/2558

Dokka 1.4.30 will (partially) fix this. There's actually nothing more we can do apart from force-closing dispatchers right now. We could remove coroutines from our code (and/or use JVM's threads), but this would be a painful, long job with the effect of degraded performance (initially Dokka was single threaded and slower by several orders of magnitude).

Giving the JVM more heap and metaspace memory and not reusing the Gradle daemon (which holds all the references that prevent the GC from doing its job) is still a viable workaround that we personally use

marcphilipp commented 3 years ago

@kamildoleglo Thanks for the update. If 1.4.30 will partially fix this, is this issue still assigned to the correct milestone?

kamildoleglo commented 3 years ago

Nope, I've forgot to do the grooming, sorry

willbuck commented 3 years ago

Thank you so much for the updates @kamildoleglo

@asarkar open source software benefits everyone, it would be great if more people paid to help support it (or better yet, understand it and contribute to it) but that sadly is not the reality we live in. You are always welcome to find and propose solutions though!

asarkar commented 3 years ago

@willbuck

it would be great if more people paid to help support it

It's a slippery slope. As soon as people start paying, there's going to be an expectation of accountability, which I'm sure will drive away many OSS contributors. I personally donate to Wikipedia, but I don't need Wikipedia to do my job; that's not the same with OSS, so when I pay for someone's time, I'll fully expect issues to be handled in a timely fashion and with a specific ETA.

I've contributed to several OSS myself, and also maintain some in my free time. I'd not want someone paying $5 start owning my time.

Anyway, I digress.

ZacSweers commented 3 years ago

@kamildoleglo the 1.4.30 release notes don't mention this, did any specific fixes for this make it in?

kamildoleglo commented 3 years ago

Yup, https://github.com/Kotlin/dokka/commit/b298da4ce1a55a6f69e7ed00020ed3af558f3750 should be in

willbuck commented 3 years ago

I'm still seeing the issue when simply running ./gradlew clean docs, but --no-daemon seems to be the only flag required to get it to complete now! Thanks for all your work @kamildoleglo

bkahlert commented 3 years ago

Can confirm the issue remains with Dokka 1.4.30 Luckily org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m works.

fabianishere commented 3 years ago

We are also experiencing the same issues on a multi-module project with quite a few large dependencies (e.g., Hadoop). Even with the max metaspace size set to 1G, we get an OutOfMemory error.

Perhaps it might help if I share a memory dump of the crash: https://drive.google.com/file/d/1Uc6_loRGXW1qxrB6kV4C92tdkbTc2Zdf/view?usp=sharing

I took a quick look at it and it seems like Dokka is trying to load all jar files of the project into memory (I suppose for analysis), but does not release references to them. That might also explain why Dokka is running out of metaspace as opposed to heap space.

CommanderTvis commented 3 years ago

Can confirm the issue remains with Dokka 1.4.30 Luckily org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m works.

Recently, we've set it to 2G to make it work 😭

chrisr3 commented 2 years ago

Hi, I have just tried upgrade from Gradle 6.9 to Gradle 7.1.1 but our Dokka 0.10.1 plugin is failing with:

* What went wrong:
Some problems were found with the configuration of task ':dokka' (type 'DokkaTask').
  - In plugin 'org.jetbrains.dokka' type 'org.jetbrains.dokka.gradle.DokkaTask' property 'configuration.collectKotlinTasks' is missing an input or output annotation.

    Reason: A property without annotation isn't considered during up-to-date checking.

    Possible solutions:
      1. Add an input or output annotation.
      2. Mark it as @Internal.

    Please refer to https://docs.gradle.org/7.1.1/userguide/validation_problems.html#missing_annotation for more details about this problem.
  - In plugin 'org.jetbrains.dokka' type 'org.jetbrains.dokka.gradle.DokkaTask' property 'configuration.name' is missing an input or output annotation.

    Reason: A property without annotation isn't considered during up-to-date checking.

    Possible solutions:
      1. Add an input or output annotation.
      2. Mark it as @Internal.

    Please refer to https://docs.gradle.org/7.1.1/userguide/validation_problems.html#missing_annotation for more details about this problem.
  - In plugin 'org.jetbrains.dokka' type 'org.jetbrains.dokka.gradle.DokkaTask' property 'dokkaRuntime' is missing an input or output annotation.

    Reason: A property without annotation isn't considered during up-to-date checking.

    Possible solutions:
      1. Add an input or output annotation.
      2. Mark it as @Internal.

Unfortunately, upgrading to Dokka 1.4.30 breaks due to OOM, and even 2GB of Metaspace isn't enough to avoid that :exploding_head:.

Seeing as the latest Dokka plugin has an apparently unfixable memory leak, can someone release a 0.10.2 version that is compatible with Gradle 7.1 please?

AlexisRiksman-TomTom commented 2 years ago

As this issue is open for 16 months now, I assume it is a tough one to solve. Therefore we took the pragmatic approach and tried to work around the problem. The development project that I am working on is growing and these out-of-memory issues with Dokka are getting more and more annoying.

With the information of other people that have investigated the issue, see previous posts on this ticker, we found two ways to work around this problem.

The daemon-kill approach is quite rude, but it scales nicely because Gradle detects when the daemon uses an excessive amount of memory. The many-partial approach is more structured, but finding the best clustering of dokka-partial tasks can be a tedious job, which may need to be redone when the project grows or gets reorganised. Also this approach takes more execution time. Gradle sometimes cannot always use the optimal amount of worker threads, but also each gradle invocation will redo the gradle configuration phase.

For more details of each work-around see my next posts on this ticket.

AlexisRiksman-TomTom commented 2 years ago

The daemon-kill approach.

Gradle was run with a single command ./gradlew docs, a custom task that depends on dokkaHtmlMultiModule, which depends on dokkaHtmlPartial for each individual module.

In the work-around we run this as:

./gradlew --stop
while true; do ./gradlew --status | grep STOPREQUESTED && ./gradlew --stop ; sleep 10 ; done &
./gradlew docs || ./gradlew docs || ./gradlew docs || ./gradlew docs || ./gradlew docs
kill %1
./gradlew --stop
AlexisRiksman-TomTom commented 2 years ago

The many-partial approach.

Before running the final ./gradlew docs command, we run a bunch of ./gradlew docsPartial commands. Each command spawns a one-time gradle daemon, but is small enough to run without memory problems in this daemon.

In the work-around we run this as:

./gradlew --stop
./gradlew docsPartial -PdocsPartialPrefix=api_ --no-daemon
./gradlew docsPartial -PdocsPartialPrefix=core_ --no-daemon
./gradlew docsPartial -PdocsPartialPrefix=media_ --no-daemon
./gradlew docsPartial -PdocsPartialPrefix=stock_ --no-daemon
./gradlew docs --no-daemon

The docsPartial task is added to the build.gradle.kts file as follows:

tasks.register("docsPartial") {
    description = "Generate intermediate documentation for modules who's name starts with" +
        " property 'docsPartialPrefix'"
    group = "Documentation"

    (findProperty("docsPartialPrefix") as String?)?.let { prefix ->
        println("docsPartialPrefix: $prefix")
        subprojects.filter { it.name.startsWith(prefix) }.forEach { module ->
            module.tasks.findByName("dokkaHtmlPartial")?.let { task ->
                dependsOn(":${module.name}:dokkaHtmlPartial")
            }
        }
    } ?: doFirst {
        throw GradleException(
            "You must set property 'docsPartialPrefix' when calling this task."
        )
    }
}
ansman commented 2 years ago

This should be fixable now that Coroutines 1.6.0 allows you to shutdown the dispatchers.

IgnatBeresnev commented 2 years ago

After some investigation, I can see that a major leak seems to have been fixed in 1.6.10 (probably somewhere on the compiler's side, verifying it now): now metaspace gets collected and classes get unloaded properly.

Before (with 1G Metaspace)

After (with 500M Metaspace)

It would help if anyone who was having problems with this could update to 1.6.10 and verify that it's gone, thanks in advance!

This should be fixable now that Coroutines 1.6.0 allows you to shutdown the dispatchers.

I've also published a dev build for coroutines dispatchers shutdown (#2325): 1.6.10-dev-144. Doesn't seem to make any noticable difference in my cases though. To use it, you have to include our maven repository:

repositories {
    maven("https://maven.pkg.jetbrains.space/kotlin/p/dokka/dev")
}
jpd236 commented 2 years ago

Unfortunately, at least in my project, 1.6.10 does not seem to have fully resolved the problem.

These iterations take 2-3x longer than with the workaround in place, and after a couple, I see "Expiring Daemon because JVM heap space is exhausted" warnings. So (to my untrained eye) it seems like metaspace usage is increasing with each run.

With the workaround still in place, this was significantly faster and could be executed multiple times without causing the running daemon to expire.

Apologies if this is something different, but the workaround has been the same for me as the one mentioned here.

IgnatBeresnev commented 2 years ago

@jpd236 actually, I'm not sure why you have problem in the first place :) Your project is rather small and for me the build finishes in under 1 minute and occupies less than 150M of Metaspace. So even if Metaspace leaks, it shouldn't affect you, since not a single GC will happen. I've cloned your project, removed -XX:MaxMetaspaceSize=512m and can reliably build it with dokka 1.5.0. Does it fail for you sometimes if you remove it?

Moreover, I've benchmarked the build time of your project with gradle-profiler and (for 1.5.0 + MaxMetaspaceSize, 1.6.0 + MaxMetaspaceSize and plain 1.6.10) it's basically the same (running with 6 warmups and 10 measured builds), median is around 24-27 seconds. csv results here

So the rest of the problems seem to be something else. "Expiring Daemon because JVM heap space is exhausted" - complains about heap, it's a different region to metaspace, so most likely unrelated. Gradle doesn't free up memory straight away after execution, daemons keep it in order to make subsequent builds faster. Like 2-3x times faster :) Daemons die when your system is low on memory, so they free up space, probably that's why you're getting the message.

You can try killing all daemons (something like pkill -f gradle.*daemon) and try running your build with ./gradlew clean dokkaHtml --no-daemon, you'll probably end up with the same execution time.

IgnatBeresnev commented 2 years ago

Note for the Metaspace leak fix in 1.6.10

Dokka loads a bunch of classes (especially for large multi-module projects), so it relies heavily on Metaspace. Even if there's no leak, you may still need to increase -XX:MaxMetaspaceSize to the point where dokka is able to load all of the classes it needs to generate documentation. I'm not exactly sure on sizing, but anything up to 512M/1G should be plenty for the majority of projects. If it requires more than that, it may indicate a leak or a very big project/module, worth reporting/checking.

The difference between 1.6.0 and 1.6.10 is that for versions <= 1.6.0 classes weren't unloaded and metaspace didn't get garbage collected. So Metaspace was filling up and never freed, causing java.lang.OutOfMemoryError: Metaspace. The fix was to give it so much Metaspace memory that it would finish the build before running out of it.

For >= 1.6.10, classes are unloaded (after a module run, for instance) and metaspace gets collected regularly, so dokka needs less -XX:MaxMetaspaceSize to finish the build. If you had to set -XX:MaxMetaspaceSize to anything larger than 1GB, you may try decreasing it until it fails (maybe you'll even be able to remove it altogether, depends on the project).

1.6.10 is mostly an enabler for larger projects that coduln't be built previously (or required too much metaspace), like aws-sdk-kotlin with 260 generated modules. On 1.6.0 with -XX:MaxMetaspaceSize=1G it fails after 10 minutes of running. On 1.6.10 with -XX:MaxMetaspaceSize=500M it completes in 2.5 hours.

update: Unfortunately, it seems like the metaspace leak is fixed only if you don't have the samples configuration property set. If you do have it set and experience the leak, try removing it, should help. We'll look into solving this as well.

tasks.withType<DokkaTask>().configureEach {
    dokkaSourceSets {
        configureEach {
            samples.from("samples/basic.kt", "samples/advanced.kt")
    }
}
jpd236 commented 2 years ago

Thanks - re: https://github.com/Kotlin/dokka/issues/1405#issuecomment-1021795270, I think my problem might have been that I was using a very old JDK to build (JDK 8). I upgraded to a newer one and was not able to reproduce the problem any more. If it's not that, I'm not sure what it was... I agree it doesn't make sense that heap was filling up and the MaxMetaspaceSize flag seemed to be mitigating it, since those are two different things. Sorry for distracting from the main issue - just thought I should follow up here for reference in case anyone else hits the same problem.

AlexisRiksman-TomTom commented 2 years ago

In our project the problem disappeared after upgrading to Dokka 1.6.10. I guess this ticket can be closed now.

GlenKPeterson commented 2 years ago

Just got again this using dokka 1.6.21 with Kotlin 1.7.0.

$ gradle --version

------------------------------------------------------------
Gradle 7.4.2
------------------------------------------------------------

Build time:   2022-03-31 15:25:29 UTC
Revision:     540473b8118064efcc264694cbcaa4b677f61041

Kotlin:       1.5.31
Groovy:       3.0.9
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          11.0.15 (Private Build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1)
OS:           Linux 4.15.0-184-generic amd64
$ gradle --no-daemon clean build publish publishToMavenLocal
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.4.2/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build 

...

> Task :org-organicdesign:test-utils-basic:dokkaJavadoc
Initializing plugins
Dokka is performing: documentation for test-utils-basic
Validity check
Creating documentation models

WARN: Attempt to load key 'java.correct.class.type.by.place.resolve.scope' for not yet loaded registry

> Task :org-organicdesign:test-utils-basic:dokkaJavadoc

> Task :org-organicdesign:test-utils-basic:dokkaJavadoc FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':org-organicdesign:test-utils-basic:dokkaJavadoc'.
> Metaspace

If I build again without making any changes, it usually works. I have org.gradle.jvmargs="-XX:MaxMetaspaceSize=1024m" in my root project gradle.properties.kts.

GlenKPeterson commented 2 years ago

Just failed again, differently:

> Task :org-organicdesign:test-utils-http:dokkaJavadoc
Initializing plugins
Dokka is performing: documentation for test-utils-http
Validity check
Creating documentation models

WARN: Attempt to load key 'java.correct.class.type.by.place.resolve.scope' for not yet loaded registry
WARN: Attempt to load key 'java.correct.class.type.by.place.resolve.scope' for not yet loaded registry
Exception in thread "DefaultDispatcher-worker-2" java.lang.OutOfMemoryError: Metaspace
Exception in thread "DefaultDispatcher-worker-1" java.lang.OutOfMemoryError: Metaspace
Exception in thread "DefaultDispatcher-worker-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[Dispatchers.Default, Continuation at org.jetbrains.dokka.base.translators.psi.DefaultPsiToDocumentableTranslator$DokkaPsiParser$parseClasslike$2$1$classlikes$1$invokeSuspend$$inlined$parallelMap$1$1.invokeSuspend(DefaultPsiToDocumentableTranslator.kt:19)@45ade651]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
        at kotlinx.coroutines.DispatchedTask.handleFatalException(DispatchedTask.kt:144)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:115)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.OutOfMemoryError: Metaspace
<===========--> 92% EXECUTING [35m 18s]
> IDLE
> IDLE
> :org-organicdesign:test-utils-http:dokkaJavadoc
> IDLE
> IDLE
IgnatBeresnev commented 2 years ago

Hi @GlenKPeterson! Are you using samples configuration property by any chance? 

tasks.withType<DokkaTask>().configureEach {
    dokkaSourceSets {
        configureEach {
            samples.from("samples/basic.kt", "samples/advanced.kt")
        }
    }
}

If not, I'm afraid that might be related to Gradle daemons not freeing memory. Try running your builds with --no-daemon and see if it helps.

Running pre-release Dokka 1.7.0 I can see that it's freeing up metaspace quite well for a project with >250 modules (running with --parallel and 2G Metaspace), and no major leaks are happening, certainly not as before: 2022-06-16_21-21-59

GlenKPeterson commented 2 years ago

Thanks for your response. I'm always using --no-daemon at the command line. I think having IntelliJ open causes a separate daemon to run?

The Gradle Tooling API that is used by IDEs and other tools to integrate with Gradle always uses the Gradle Daemon to execute builds. If you are executing Gradle builds from within your IDE you are using the Gradle Daemon and do not need to enable it for your environment. https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:tools_and_ides

Could that be what's messing me up?

I searched my entire project for "samples" and only found an image: sampleScreenShot.png. I'm still out of metaspace on and off.

I only have one gradle.properties.kts in the master project with org.gradle.jvmargs="-XX:MaxMetaspaceSize=1024m". I assume that's the right place for it.