Seeing different results in output reports from CLI

[randomstability]

Reporting Bugs/Errors

Java version: "1.8.0_144"

OS: Windows 7 Enterprise 64 bit Service Pack: SP1 Patch level: Current through August 2017

DependencyCheck version: 2.1.1 CLI

Example command syntax: C:\Downloads>C:\Downloads\dependency-check-2.1.1-release\dependency-check\bin\dependency-check.bat -project webgoat -s C:\Storage\webgoat-container-7.1-exec.jar -o C:\Downloads\dependency-check-reports\webgoat_7.1.1_report.html

Attached examples of varying report output: dependency-check-reports_report_varies_with_multiple_scans.zip

=============================================

This is recreatable using publicly accessible content. I attempted to scan webgoat 7.1 from https://github.com/WebGoat/WebGoat/releases/download/7.1/webgoat-container-7.1-exec.jar

I repeated the scan using the CLI several times (one at a time) over the course of 20 minutes, and the report was inconsistent in its content on random scans. The NVD database downloaded only once (the first time). Every other time, it indicated there was no need to update it.

I suggest trying to repeat my scan numerous times, each time into a new report output file. After some number of scans, I expect the report content to vary. In my case I saw differences by the 3rd scan. But it was as infrequent as 5 reports in a row.

I would recommend using a text editor with a file compare feature, like Notepad++ to compare the files and discover the differences -which appeared random to me.

[jeremylong]

I'm still researching this - but it appears to have to do with the CentralAnalyzer not reliably downloading information about the JAR files.

[stefanneuhaus]

And in fact the CentralAnalyzer is the culprit. When encountering an IOException during fetching the MavenArtifacts from MavenCentral it disables itself (errorFlag = true).

I was able to reproduce the issue: I ran analyzing my sample-project in a loop. During about 2/3 runs I encountered at least one connection issue (500):

Could not connect to Central received response code: 500 Internal Server Error
java.io.IOException: Could not connect to Central
    at org.owasp.dependencycheck.data.central.CentralSearch.searchSha1(CentralSearch.java:183)
    at org.owasp.dependencycheck.analyzer.CentralAnalyzer.analyzeDependency(CentralAnalyzer.java:215)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:137)
    at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Though not every error affected the actual result, I was still able to get unstable results. grep "^Found" analyze.log | sort | uniq -c yielded:

 177 Found 18 vulnerabilities in project sample-project: CVE-2011-0509, CVE-2012-6153, CVE-2013-4499, CVE-2014-0107, CVE-2014-0114, CVE-2014-3577, CVE-2015-3253, CVE-2015-5262, CVE-2016-10006, CVE-2016-2510, CVE-2016-3093, CVE-2016-6497, CVE-2017-5662, CVE-2017-5662, CVE-2017-5662, CVE-2017-5664, CVE-2017-5664, CVE-2017-5929
   6 Found 21 vulnerabilities in project sample-project: CVE-2011-0509, CVE-2012-6153, CVE-2013-4499, CVE-2014-0107, CVE-2014-0114, CVE-2014-3577, CVE-2015-3192, CVE-2015-3253, CVE-2015-5211, CVE-2015-5262, CVE-2016-10006, CVE-2016-2510, CVE-2016-3093, CVE-2016-5007, CVE-2016-6497, CVE-2017-5662, CVE-2017-5662, CVE-2017-5662, CVE-2017-5664, CVE-2017-5664, CVE-2017-5929
2391 Found 22 vulnerabilities in project sample-project: CVE-2011-0509, CVE-2012-6153, CVE-2013-4499, CVE-2014-0107, CVE-2014-0114, CVE-2014-3577, CVE-2015-3192, CVE-2015-3253, CVE-2015-5211, CVE-2015-5262, CVE-2016-10006, CVE-2016-2510, CVE-2016-3093, CVE-2016-5007, CVE-2016-6497, CVE-2016-9878, CVE-2017-5662, CVE-2017-5662, CVE-2017-5662, CVE-2017-5664, CVE-2017-5664, CVE-2017-5929

(These results were obtained using a version from 2017-09-23, but I obtained similar results with the current version fa377cf from this morning. Sadly I overwrote the logs.)

[randomstability]

I have returned to check on this, and it may be greatly improved in version 3.1.1, but I've still identified some small differences across multiple runs of the same scan.

Updated: Java version: "1.8.0_162"

OS: Windows 7 Enterprise 64 bit Service Pack: SP1 Patch level: Current through February 2018

DependencyCheck version: 3.1.1 CLI

========================================

The issues seem to be in the evidence section. Multiple report runs said there were 31 pieces, another set of reports said 33. I have not tested more in-depth to know if it varies by more than this one example.

Probably not critical enough to focus on yet. But just in case this indicates that the main findings could also be off, I want to report it here.

If it helps, these are the results searching for a some example strings.

Report 1: string: ">1.2.17<"

Line 12337:  <tr><td>Version</td><td>Manifest</td><td>Bundle-Version</td><td>1.2.17</td><td>High</td></tr>
Line 12388:  <tr><td>Version</td><td>central</td><td>version</td><td>1.2.17</td><td>High</td></tr>
Line 12532:  <tr><td>Version</td><td>file</td><td>version</td><td>1.2.17</td><td>Highest</td></tr>
Line 12594:  <tr><td>Version</td><td>manifest: org.apache.log4j</td><td>Implementation-Version</td><td>1.2.17</td><td>Medium</td></tr>
Line 12672:  <tr><td>Version</td><td>pom</td><td>version</td><td>1.2.17</td><td>Highest</td></tr>

string: "manifest: org.apache.log4j"

Line 10022:  <tr><td>Product</td><td>manifest: org.apache.log4j</td><td>Implementation-Title</td><td>log4j</td><td>Medium</td></tr>
Line 11565:  <tr><td>Vendor</td><td>manifest: org.apache.log4j</td><td>Implementation-Vendor</td><td>&quot;Apache Software Foundation&quot;</td><td>Medium</td></tr>
Line 12594:  <tr><td>Version</td><td>manifest: org.apache.log4j</td><td>Implementation-Version</td><td>1.2.17</td><td>Medium</td></tr>

Report 2: string: ">1.2.17<" - Missing the two "manifest" entries.

Line 12388:  <tr><td>Version</td><td>central</td><td>version</td><td>1.2.17</td><td>High</td></tr>
Line 12532:  <tr><td>Version</td><td>file</td><td>version</td><td>1.2.17</td><td>Highest</td></tr>
Line 12672:  <tr><td>Version</td><td>pom</td><td>version</td><td>1.2.17</td><td>Highest</td></tr>

string: "manifest: org.apache.log4j" - missing the Version entry.

Line 10022:  <tr><td>Product</td><td>manifest: org.apache.log4j</td><td>Implementation-Title</td><td>log4j</td><td>Medium</td></tr>
Line 11565:  <tr><td>Vendor</td><td>manifest: org.apache.log4j</td><td>Implementation-Vendor</td><td>&quot;Apache Software Foundation&quot;</td><td>Medium</td></tr>

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.