Performance Degradation On Large Multi-Projects With Nested Transitive Project Dependencies

[mroeppis]

Observed after moving from 6.5.3 to 7.0.0.

Fault Description

Running ODC against a large Gradle project with several hundred sub-projects having nested cross-references (i.e. sub A depends on sub B which again depends on sub C and so on) results in very long execution times. That makes it difficult to use the plugin for a quick check (e.g. after adjusting some vulnerability exposing dependencies).

As a reference: I'm used to see ODC complete after 3 minutes. Now I have to wait for more than 30 minutes (not including DB update time).

I attach mwe.zip for reproduction. Tested with Gradle 7.1.

Debugging into Depedency Check Core Engine you can see the generated workload which leads to increased execution time for the following analyzers.

Analyzer	Analysis Task Count v6.5.3	Analysis Task Count v7.0.0	Mean Execution time [s] v6.5.3	Mean Execution time [s] v7.0.0
Dependency Bundling Analyzer	496	105,607	< 1	1,900
Dependency Merging Analyzer			< 1	90
CPE Analyzer			2	52
Sonatype OSS Index Analyzer			< 1	30

While different platforms may result in different execution times the point here is that we have an increase by at least factor 10 on a fixed platform. This behavior hasn't changed since version 7.0.0.

Analysis

The cause seems to be a change in AbstractAnalyze#addDependency(). See #239, #245 and commit 5f512b4 for reference.

https://github.com/dependency-check/dependency-check-gradle/blob/fed02e4456a3dc6c6cbd7ae9ace51574c7ac69e2/src/main/groovy/org/owasp/dependencycheck/gradle/tasks/AbstractAnalyze.groovy#L526-L530

Judging from the issue description

The false positives seem to arise because the plugin has lost the link from a given dependency jar to the actual Maven package it came from.

I would say the MavenArtifact creation is correct since it implicitly adds the package URL to the dependency. Always adding the dependency to the list of dependencies from the Engine, however, may create duplicates because it might already be in the list.

Suggested Fix

Use the former approach to check whether a dependency already exists. If so, update solely the project reference since the only reason for a recurring dependency is a different project or configuration.

[jeremylong]

Thank you for the sample project and analysis. The issue is caused by adding duplicate virtual dependencies - in the case of the project dependencies that were not built and as such do not have an associated JAR file. This likely affects the Maven integration as well.

Test 1 (completes in 30+ minutes ):

$ ./gradlew clean
$ ./gradlew dependencyCheckAggregate

Test 2 (completes in ~1 minute):

$ ./gradlew clean
$ ./gradlew build dependencyCheckAggregate

[mroeppis]

@jeremylong : Thx 4 resolving this so quickly. Just tried out the new version (7.4.0). Way better!