Dependency Graph missing transitive dependencies when dependency has multiple sources

[msymons]

When cyclonedx-maven-plugin 2.5.1 generates a BOM for a project that has direct dependencies X and Y and X and Y both have Z as a transitive dependency, then the dependency graph will describe Z as only being a dependency of X and will ignore it also being a dependency of Y.

This causes problems when Z has a vulnerability and X has an update available that contains a fixed version of Z, but Y does not. The dependency graph would suggest that updating X is sufficient to deal with the vulnerability. But it is not... and only AFTER the upgrade is completed and a new BOM generated will the dependency graph now show the dependency of Y on Z.

The problem can be illustrated by setting up a simple project with the following two dependencies

<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-broker</artifactId>
    <version>5.16.2</version>
</dependency>
<dependency>
    <groupId>org.apache.cassandra</groupId>
    <artifactId>cassandra-all</artifactId>
    <version>4.0.0</version>
</dependency>

Both have com.fasterxml.jackson.core:jackson-databind 2.9.10.8 as a dependency.

In plugin 2.5.1 BOM graph, jackson-databind is only reported as being a dependency of activemq-broker

[stevespringett]

The output from mvn dependency:tree shows the following which seems to be the same as what CycloneDX is reporting.

com.example:issue-116:jar:1.0.0
+- org.apache.activemq:activemq-broker:jar:5.16.2:compile
|  +- org.apache.activemq:activemq-client:jar:5.16.2:compile
|  |  +- org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1:compile
|  |  +- org.fusesource.hawtbuf:hawtbuf:jar:1.11:compile
|  |  \- org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:jar:1.0.1:compile
|  +- org.apache.activemq:activemq-openwire-legacy:jar:5.16.2:compile
|  \- com.fasterxml.jackson.core:jackson-databind:jar:2.9.10.8:compile
\- org.apache.cassandra:cassandra-all:jar:4.0.0:compile
   +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
   +- org.lz4:lz4-java:jar:1.7.1:compile
   +- com.google.guava:guava:jar:27.0-jre:compile
   +- commons-cli:commons-cli:jar:1.1:compile
   +- commons-codec:commons-codec:jar:1.9:compile
   +- org.apache.commons:commons-lang3:jar:3.11:compile
   +- org.apache.commons:commons-math3:jar:3.2:compile
   +- org.antlr:ST4:jar:4.0.8:compile
   +- org.antlr:antlr-runtime:jar:3.5.2:compile
   +- org.slf4j:slf4j-api:jar:1.7.25:compile
   +- org.slf4j:log4j-over-slf4j:jar:1.7.25:compile
   +- org.slf4j:jcl-over-slf4j:jar:1.7.25:compile
   +- com.fasterxml.jackson.core:jackson-core:jar:2.9.10:compile
   +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.10:compile
   +- com.googlecode.json-simple:json-simple:jar:1.1:compile
   +- com.boundary:high-scale-lib:jar:1.0.6:compile
   +- org.yaml:snakeyaml:jar:1.26:compile
   +- org.mindrot:jbcrypt:jar:0.3m:compile
   +- io.airlift:airline:jar:0.8:compile
   +- io.dropwizard.metrics:metrics-core:jar:3.1.5:compile
   +- io.dropwizard.metrics:metrics-jvm:jar:3.1.5:compile
   +- io.dropwizard.metrics:metrics-logback:jar:3.1.5:compile
   +- com.addthis.metrics:reporter-config3:jar:3.0.3:compile
   |  \- com.addthis.metrics:reporter-config-base:jar:3.0.3:compile
   +- com.clearspring.analytics:stream:jar:2.5.2:compile
   +- ch.qos.logback:logback-core:jar:1.2.3:compile
   +- ch.qos.logback:logback-classic:jar:1.2.3:compile
   +- net.java.dev.jna:jna:jar:5.6.0:compile
   +- com.github.jbellis:jamm:jar:0.3.2:compile
   +- io.netty:netty-all:jar:4.1.58.Final:compile
   +- net.openhft:chronicle-queue:jar:5.20.123:compile
   +- net.openhft:chronicle-core:jar:2.20.126:compile
   +- net.openhft:chronicle-bytes:jar:2.20.111:compile
   +- net.openhft:chronicle-wire:jar:2.20.117:compile
   +- net.openhft:chronicle-threads:jar:2.20.111:compile
   +- org.fusesource:sigar:jar:1.6.4:compile
   +- org.eclipse.jdt.core.compiler:ecj:jar:4.6.1:compile
   +- org.caffinitas.ohc:ohc-core:jar:0.5.1:compile
   +- org.caffinitas.ohc:ohc-core-j8:jar:0.5.1:compile
   +- com.github.ben-manes.caffeine:caffeine:jar:2.3.5:compile
   +- org.jctools:jctools-core:jar:3.1.0:compile
   +- org.ow2.asm:asm:jar:7.1:compile
   +- com.carrotsearch:hppc:jar:0.8.1:compile
   +- org.gridkit.jvmtool:sjk-cli:jar:0.14:compile
   +- org.gridkit.jvmtool:sjk-core:jar:0.14:compile
   +- org.gridkit.jvmtool:sjk-stacktrace:jar:0.14:compile
   +- org.gridkit.jvmtool:mxdump:jar:0.14:compile
   +- org.gridkit.lab:jvm-attach-api:jar:1.5:compile
   +- com.beust:jcommander:jar:1.30:compile
   +- org.gridkit.jvmtool:sjk-json:jar:0.14:compile
   +- com.github.luben:zstd-jni:jar:1.3.8-5:compile
   +- org.psjava:psjava:jar:0.1.19:compile
   +- io.netty:netty-tcnative-boringssl-static:jar:2.0.36.Final:compile
   +- javax.inject:javax.inject:jar:1:compile
   +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
   +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
   +- de.jflex:jflex:jar:1.8.2:compile
   |  \- com.github.vbmacher:java-cup-runtime:jar:11b-20160615:compile
   +- com.github.rholder:snowball-stemmer:jar:1.3.0.581.1:compile
   \- com.googlecode.concurrent-trees:concurrent-trees:jar:2.4.0:compile

CycloneDX Maven plugin leverages org.apache.maven.shared.dependency.graph.DependencyGraphBuilder to generate the graph. If this is a defect, then the issue is with Maven itself.

[stevespringett]

Blocked by https://issues.apache.org/jira/browse/MDEP-644

[msymons]

I have performed testing using depgraph-maven-plugin and found that it generates a graph that does report the duplicate versions.

The following will generate text (output to console) for a project that has no module (tweak goal for multi-module project).

<plugin>
    <groupId>com.github.ferstl</groupId>
    <artifactId>depgraph-maven-plugin</artifactId>
    <version>3.3.0</version>
    <executions>
        <execution>
            <id>graph</id>
            <phase>package</phase>
            <goals>
                <goal>graph</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <graphFormat>text</graphFormat>
        <showConflicts>true</showConflicts>
        <showDuplicates>true</showDuplicates>
        <showVersions>true</showVersions>
    </configuration>
</plugin>

The results, tested using a project with a single component org.gridkit.jvmtool:sjk-core:0.14 look like this:

 [INFO] Dependency graph:
 maven-tests:0.2-SNAPSHOT:compile
 \- sjk-core:0.14:compile
    +- sjk-cli:0.14:compile
    |  \- jcommander:1.30:compile
    +- sjk-stacktrace:0.14:compile
    |  \- sjk-json:0.14:compile
    +- sjk-hflame:0.14:compile
    |  \- sjk-stacktrace:0.14:compile (omitted for duplicate)
    +- sjk-jfr5:0.5:compile
    +- sjk-jfr6:0.7:compile
    +- sjk-jfr-standalone:0.7:compile
    |  \- sjk-stacktrace:0.14:compile (omitted for conflict: 0.13)
    +- sjk-nps:0.5:compile
    \- jvm-attach-api:1.5:compile

Note the "omitted for duplicate". That's what is currently missing from our SBOMs. (Note also that groupIds are not listed.. but can be, via config).

Compare with {{dependency:tree}} output:

 [INFO] --- maven-dependency-plugin:3.2.0:tree (default-cli) @ maven-tests ---
 [INFO] net.weareact.maven-tests:maven-tests:pom:0.2-SNAPSHOT
 [INFO] \- org.gridkit.jvmtool:sjk-core:jar:0.14:compile
 [INFO]    +- org.gridkit.jvmtool:sjk-cli:jar:0.14:compile
 [INFO]    |  \- com.beust:jcommander:jar:1.30:compile
 [INFO]    +- org.gridkit.jvmtool:sjk-stacktrace:jar:0.14:compile
 [INFO]    |  \- org.gridkit.jvmtool:sjk-json:jar:0.14:compile
 [INFO]    +- org.gridkit.jvmtool:sjk-hflame:jar:0.14:compile
 [INFO]    +- org.perfkit.sjk.parsers:sjk-jfr5:jar:0.5:compile
 [INFO]    +- org.perfkit.sjk.parsers:sjk-jfr6:jar:0.7:compile
 [INFO]    +- org.perfkit.sjk.parsers:sjk-jfr-standalone:jar:0.7:compile
 [INFO]    +- org.perfkit.sjk.parsers:sjk-nps:jar:0.5:compile
 [INFO]    \- org.gridkit.lab:jvm-attach-api:jar:1.5:compile

One thing that is interesting is that sjk-core is a dependency of cassandra-all (see original description) and yet does not report it's dependencies in that context (Maven dependency:tree). I am pretty sure that this is related to the maven warnings:

[WARNING] The POM for org.perfkit.sjk.parsers:sjk-jfr5:jar:0.5 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.perfkit.sjk.parsers:sjk-jfr6:jar:0.7 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.perfkit.sjk.parsers:sjk-nps:jar:0.5 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

[msymons]

Although it is not relevent to the issue of reporting duplicate versions, I also did some testing to generate PlantUML. Thus for the same component sjk-core:

           <plugin>
               <groupId>com.github.ferstl</groupId>
               <artifactId>depgraph-maven-plugin</artifactId>
               <version>3.3.0</version>
               <executions>
                   <execution>
                       <id>graph</id>
                       <phase>package</phase>
                       <goals>
                           <goal>graph</goal>
                       </goals>
                   </execution>
               </executions>
               <configuration>
                   <graphFormat>puml</graphFormat>
                   <outputDirectory>target</outputDirectory>
                   <showConflicts>true</showConflicts>
                   <showDuplicates>true</showDuplicates>
                   <showVersions>true</showVersions>
               </configuration>
           </plugin>

The resulting output is:

@startuml
skinparam defaultTextAlignment center
skinparam rectangle {
  BackgroundColor<<optional>> beige
  BackgroundColor<<test>> lightGreen
  BackgroundColor<<runtime>> lightBlue
  BackgroundColor<<provided>> lightGray
}
rectangle "sjk-cli\n0.14" as org_gridkit_jvmtool_sjk_cli_jar
rectangle "jcommander\n1.30" as com_beust_jcommander_jar
rectangle "sjk-core\n0.14" as org_gridkit_jvmtool_sjk_core_jar
rectangle "sjk-stacktrace\n0.14" as org_gridkit_jvmtool_sjk_stacktrace_jar
rectangle "sjk-json\n0.14" as org_gridkit_jvmtool_sjk_json_jar
rectangle "sjk-hflame\n0.14" as org_gridkit_jvmtool_sjk_hflame_jar
rectangle "sjk-jfr5\n0.5" as org_perfkit_sjk_parsers_sjk_jfr5_jar
rectangle "sjk-jfr6\n0.7" as org_perfkit_sjk_parsers_sjk_jfr6_jar
rectangle "sjk-jfr-standalone\n0.7" as org_perfkit_sjk_parsers_sjk_jfr_standalone_jar
rectangle "sjk-nps\n0.5" as org_perfkit_sjk_parsers_sjk_nps_jar
rectangle "jvm-attach-api\n1.5" as org_gridkit_lab_jvm_attach_api_jar
rectangle "maven-tests\n0.2-SNAPSHOT" as net_weareact_maven_tests_maven_tests_pom
org_gridkit_jvmtool_sjk_cli_jar -[#000000]-> com_beust_jcommander_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_gridkit_jvmtool_sjk_cli_jar
org_gridkit_jvmtool_sjk_stacktrace_jar -[#000000]-> org_gridkit_jvmtool_sjk_json_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_gridkit_jvmtool_sjk_stacktrace_jar
org_gridkit_jvmtool_sjk_hflame_jar .[#D3D3D3].> org_gridkit_jvmtool_sjk_stacktrace_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_gridkit_jvmtool_sjk_hflame_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_perfkit_sjk_parsers_sjk_jfr5_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_perfkit_sjk_parsers_sjk_jfr6_jar
org_perfkit_sjk_parsers_sjk_jfr_standalone_jar .[#FF0000].> org_gridkit_jvmtool_sjk_stacktrace_jar: 0.13
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_perfkit_sjk_parsers_sjk_jfr_standalone_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_perfkit_sjk_parsers_sjk_nps_jar
org_gridkit_jvmtool_sjk_core_jar -[#000000]-> org_gridkit_lab_jvm_attach_api_jar
net_weareact_maven_tests_maven_tests_pom -[#000000]-> org_gridkit_jvmtool_sjk_core_jar
@enduml

If you do not have plantuml-visualizer browser extension installed (available for both Firefox and Chrome, then this is what the above looks like when rendered:

Note that the 3 components that Maven had a problem with (sjk-jfr5, etc) are ALL included!

[7-bit]

Hi @stevespringett,

is this issue still blocked because of the linked Jira issue? If I understand the Jira issue correctly it is fixed.

[stevespringett]

The fix in the JIRA issue doesn't work. It's also not blocked since depgraph-maven-plugin seems to generate a proper graph. So work needs to be done to investigate why it works and port the working portion back to this plugin.

[ThomGeG]

I've stumbled across this after also experiencing this issue and another of entirely missing dependencies, so I've taken a crack at fixing it.

sschuberth was kind enough to link this issue to https://github.com/oss-review-toolkit/ort/issues/4720, which contains details on the fix for the problem. Basically when in verbose mode dependency:tree uses a different builder than normal, so we just need to follow suit.

I've already made the changes to switch to the different type of builder on a fork (here) that ensures we list our duplicate dependencies, however it's also introduced a new problem with projects set to package WARs. In such cases the dependency graph is missing entirely from the BOM because the root node doesn't have any children.

I'm still in the middle of trying to figure out what's wrong, hopefully it won't take me longer than a day or two to fix whatever it is and get a PR raised.

[ThomGeG]

Alright, I'm back after finding the problem and it's unfortunately in the org.apache.maven.shared:maven-dependency-tree being used:

• https://issues.apache.org/jira/browse/MSHARED-994
• https://github.com/apache/maven-dependency-tree/pull/7/files

The fix has thankfully already been made back in December last year and I've confirmed it resolves our issue of no dependency graph in the BOMs, however they haven't done a release since way back in July of 2021.

I'm not too familiar with how the Apache ecosystem works, so it'll take me some time to figure out their process and try get them to do a release. If anybody else is more familiar with it, feel free to let me know or take a crack yourself

[ThomGeG]

Alright, I've emailed the Apache dev mailing list asking if somebody can release the latest version of maven-dependency-tree so we can move to it (hopefully I've done it right). Apparently releases are something only a PMC can do and they're quite an involved process, so if somebody is kind enough to help us it would still be a while before we can get our hands on it.

In the meantime I've also had a look at the degraph-maven-plugin Steve and Mark were previously talking about. It looks like this is the bit of code actually grabbing the dependency graph before walking it, which is just directly using the same classes from maven that maven-dependency-tree uses, so they're bypassing the problem.

It's probably also worth mentioning that the -Dverbose parameter on dependency:tree breaking trees for WARs is only a 'recent' issue. Part of why it took me so long to find the problem was that I was actually using a much older version of maven-dependency-plugin (and therefore maven-dependency-tree) for some testing which led me astray. Back in at least v2.8 the TreeMojo worked differently and would at least build trees.

---

If nothing comes of Apache releasing the latest version of maven-dependency-tree, I'll take a crack at bypassing it like degraph-maven-plugin has done and raise a PR.

[stevespringett]

@ThomGeG Thank you so much for the amazing investigative work in tracking down the root cause and multiple solutions. Yes, Apache release process is a bit involved.