[Question / Bug] dependencyCheckAggregate task is stuck

[alixwar]

Problem: My Gradle project times out (1 hour execution time) whereas some of my other Gradle projects, with same/similar setup, execute within 5 minutes or so. There's no obvious difference in configuration. The build is stuck in this phase in one of the projects:

- Analyzing main:debugCompileClasspath
WARNING: step_script could not run to completion because the timeout was exceeded. For more control over job and script timeouts see: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#set-script-and-after_script-timeouts
ERROR: Job failed: execution took longer than 1h0m0s seconds

What could I do to troubleshoot this? I don't think changing the timeout will help as I've run without timeout and waited 2 hours and counting...

Top level declaration (Gradle Kotlin DSL):

    extensions.findByType(DependencyCheckExtension::class)?.let {
        it.formats = listOf(HTML.name, XML.name)
        it.nvd.maxRetryCount = 0
        it.nvd.apiKey = <HIDDEN>
    }

The task that I run:

gradle dependencyCheckAggregate --info

(I've also tried with --stacktrace but with no additional information...) My setup:

• Multi-module project
• Gradle 8.8
• JDK 21
• Dependency check version 10.0.2

Possibly related: https://github.com/jeremylong/DependencyCheck/issues/3897

[chadlwilson]

You might be stuck in a garbage collection loop due to low memory in Gradle Daemons perhaps? You could look at memory/cpunusage and see if it's stuck (doing nothing) or stuck (doing something).

As a side note, do you happen to have JavaScript dependencies in your projects?

[alixwar]

You might be stuck in a garbage collection loop due to low memory in Gradle Daemons perhaps? You could look at memory/cpunusage and see if it's stuck (doing nothing) or stuck (doing something).

As a side note, do you happen to have JavaScript dependencies in your projects?

Memory usage doesn't change. CPU doesn't change. JavaScript dependencies, not directly—however I'm using Open API generator (https://github.com/OpenAPITools/openapi-generator) for example in one of my projects in the build script to generate source code which could in turn depend on JavaScript.

I did a thread dump now after getting stuck using jstack on both involved java processes (started from command line, I'm running Windows 11 by the way):

1. org.gradle.launcher.daemon.bootstrap.GradleDaemon gradledaemon.txt

2. org.gradle.wrapper.GradleWrapperMain gradlewrappermain.txt

Maybe these are helpful to someone.

gradle-properties:

org.gradle.jvmargs=-Xmx2096m -XX:MaxMetaspaceSize=768m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
org.gradle.parallel=true
systemProp.file.encoding=UTF-8
android.useAndroidX=true
android.enableR8.fullMode=false
android.enableJetifier=false

[chadlwilson]

Looks like a deadlock of some sort to me.

Worker threads are waiting for a lock held by included builds which in turn is waiting for something else which is not happening when trying to resolve dependencies across modules.

The thread that has the lock that everything else is waiting on is at

https://github.com/dependency-check/dependency-check-gradle/blob/aa6e9d078370378167c2d2a921d4fdab0229cdb4/src/main/groovy/org/owasp/dependencycheck/gradle/tasks/AbstractAnalyze.groovy#L575

Could also be to do with issues with Gradle and/or the Gradle plugins you are using along with parallel execution more than anything within OWASP Dependency check (but I don't really know).

All I can suggest is

• Since this is while analyzing artifact types, you could try narrowing it down by excluding various artifact types from the default set https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration-aggregate.html
• disable parallel execution as a workaround, perhaps only when running ODC (--max-workers=1)
• or reduce parallelism/# workers if more of them increases the chance of it getting stuck
• try upgrading Gradle and/or various plugins to see if fixed there?
• experiment with disabling/skipping certain Gradle configurations to see if you can find one that causes problems
• experiment to see whether you get the same problem on Linux? Seems you are on Windows.

[alixwar]

Looks like a deadlock of some sort to me.

Worker threads are waiting for a lock held by included builds which in turn is waiting for something else which is not happening when trying to resolve dependencies across modules.

The thread that has the lock that everything else is waiting on is at

https://github.com/dependency-check/dependency-check-gradle/blob/aa6e9d078370378167c2d2a921d4fdab0229cdb4/src/main/groovy/org/owasp/dependencycheck/gradle/tasks/AbstractAnalyze.groovy#L575

Could also be to do with issues with Gradle and/or the Gradle plugins you are using along with parallel execution more than anything within OWASP Dependency check (but I don't really know).

All I can suggest is

• Since this is while analyzing artifact types, you could try narrowing it down by excluding various artifact types from the default set https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration-aggregate.html
• disable parallel execution as a workaround, perhaps only when running ODC (--max-workers=1)
• or reduce parallelism/# workers if more of them increases the chance of it getting stuck
• try upgrading Gradle and/or various plugins to see if fixed there?
• experiment with disabling/skipping certain Gradle configurations to see if you can find one that causes problems
• experiment to see whether you get the same problem on Linux? Seems you are on Windows.

Thanks for the suggestions. I'll update this reply with new information as I collect it:

Since this is while analyzing artifact types, you could try narrowing it down by excluding various artifact types from the default set https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration-aggregate.html

This does not seem to help. I've even tried with an empty list but it still gets stuck

disable parallel execution as a workaround, perhaps only when running ODC (--max-workers=1)

I've tried this but it doesn't work (neither Windows nor MacOS), see findings below.

or reduce parallelism/# workers if more of them increases the chance of it getting stuck

I'm not sure exactly what this means, but I am guessing that it is not feasible for my project where parallelism is a must to reduce build time

try upgrading Gradle and/or various plugins to see if fixed there?

I am running Gradle 8.7 due to https://github.com/google/play-services-plugins/issues/299 but I am pretty certain that I had the same issue when I was briefly on 8.8/8.9 before reverting

experiment with disabling/skipping certain Gradle configurations to see if you can find one that causes problems

I'm not sure exactly what this means

experiment to see whether you get the same problem on Linux? Seems you are on Windows.

This is a problem both on Windows and MacOS (M1) at least

Interesting finding 1 Although I have explicitly skipped a sub project, let's call it "x":

it.skipProjects =
            listOf(":x")

I still get stuck on

:x:dependencyCheckAggregate

Interesting finding 2 Observed in Windows (11). I see several of these entries in the output log, even with --max-workers=1:

NVD API request failures are occurring; retrying request for the 1 time

The number never changes, but I can see the same log message several times in quick succession

Interesting finding 3 A crash when running with --max-workers=1 on MacOS (on Windows I don't see this crash at all, but it also gets stuck):

- Analyzing main:debugCompileClasspath
Exception in thread "Worker-1" java.lang.NoClassDefFoundError: org/eclipse/osgi/framework/log/FrameworkLogEntry
    at org.eclipse.core.internal.runtime.PlatformLogWriter.getLog(PlatformLogWriter.java:88)
    at org.eclipse.core.internal.runtime.PlatformLogWriter.logging(PlatformLogWriter.java:47)
    at org.eclipse.core.internal.runtime.RuntimeLog.log(RuntimeLog.java:100)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:91)
Caused by: java.lang.ClassNotFoundException: org.eclipse.osgi.framework.log.FrameworkLogEntry
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
    at com.diffplug.spotless.FeatureClassLoader.findClass(FeatureClassLoader.java:79)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
    ... 4 more

After this execution does not terminate but there is no more log output

Interesting finding 4 I am on Kotlin 2.0 (necessary because of dependency androidx.compose.material3.adaptive:adaptive). Possibly it could be part of the issue?

[chadlwilson]

Api request failures have nothing to do with deadlocks. Plenty of other tickets and guidance on this one.

[alixwar]

Api request failures have nothing to do with deadlocks. Plenty of other tickets and guidance on this one.

Sure, but that is just a small finding while I attempted to troubleshoot. I have other projects where the plugin works well with the same setup (difference in dependencies and modular structure of course)