search

jeremylong / DependencyCheck

OWASP dependency-check is a software composition analysis utility that detects publicly disclosed vulnerabilities in application dependencies.
https://owasp.org/www-project-dependency-check/
Apache License 2.0
6.44k stars 1.28k forks source link

NPE from OssIndexAnalyzer fetching component-report #4527

Open OrangeDog opened 2 years ago

OrangeDog commented 2 years ago

Describe the bug

[WARNING] Failed to fetch component-report for: pkg:maven/io.netty/netty-handler@4.1.77.Final
java.lang.NullPointerException
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform (OssIndexAnalyzer.java:350)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5 (OssIndexAnalyzer.java:256)
    at java.util.stream.ReferencePipeline$3$1.accept (ReferencePipeline.java:195)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining (ArrayList.java:1655)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:484)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:474)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential (ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential (ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEachOrdered (ReferencePipeline.java:502)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.enrich (OssIndexAnalyzer.java:257)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency (OssIndexAnalyzer.java:163)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:131)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
    at java.lang.Thread.run (Thread.java:829)

Version of dependency-check used dependency-check-maven 7.1.0

Log file https://gist.github.com/OrangeDog/9f976a83dd5af51f51ce677944384420

To Reproduce Unsure. netty-handler is included via io.lettuce:lettuce-core:6.1.8.RELEASE

Expected behavior No NPE warnings in log.

OrangeDog commented 2 years ago

In another project I get the same NPEs for the following:

pkg:maven/commons-httpclient/commons-httpclient@3.1
pkg:maven/com.fasterxml.woodstox/woodstox-core@5.2.1
pkg:maven/org.thymeleaf.extras/thymeleaf-extras-springsecurity5@3.0.4.RELEASE
NearW commented 2 years ago

Same here for the following in our project:

pkg:maven/org.bouncycastle/bcprov-ext-jdk15on@1.69
pkg:maven/io.github.classgraph/classgraph@4.8.108
pkg:maven/org.bouncycastle/bcprov-jdk15on@1.69
pkg:maven/io.netty/netty-handler@4.1.77.Final
somera commented 2 years ago

Same for

pkg:maven/com.h2database/h2@2.1.210
pkg:maven/com.google.guava/guava@31.0.1-jre
pkg:maven/com.google.guava/guava@31.1-jre
pkg:maven/org.bouncycastle/bcprov-jdk15on@1.68

And

17:03:04  [INFO] --- dependency-check-maven:7.1.0:aggregate (default-cli) @ discovery ---
17:03:06  [INFO] Checking for updates
17:03:06  [INFO] Skipping NVD check since last check was within 4 hours.
17:03:06  [INFO] Skipping RetireJS update since last update was within 24 hours.
17:03:06  [INFO] Check for updates complete (12 ms)

...

17:03:06  [INFO] Analysis Started
17:03:07  [INFO] Finished Archive Analyzer (0 seconds)
17:03:07  [INFO] Finished File Name Analyzer (0 seconds)
17:03:07  [INFO] Finished Jar Analyzer (0 seconds)
17:03:07  [INFO] Finished Dependency Merging Analyzer (0 seconds)
17:03:07  [INFO] Finished Version Filter Analyzer (0 seconds)
17:03:07  [INFO] Finished Hint Analyzer (0 seconds)
17:03:08  [INFO] Created CPE Index (1 seconds)
17:03:10  [INFO] Finished CPE Analyzer (2 seconds)
17:03:10  [INFO] Finished False Positive Analyzer (0 seconds)
17:03:10  [INFO] Finished NVD CVE Analyzer (0 seconds)
17:03:10  00:00  INFO: Vulnerability found: bootstrap below 3.4.1
17:03:10  00:00  INFO: Vulnerability found: bootstrap below 3.4.0
17:03:10  00:00  INFO: Vulnerability found: bootstrap below 3.4.0
17:03:10  00:00  INFO: Vulnerability found: bootstrap below 3.4.0
17:03:10  00:00  INFO: Vulnerability found: bootstrap below 3.4.0
17:03:10  [INFO] Finished RetireJS Analyzer (0 seconds)
17:03:10  [WARNING] Failed to fetch component-report for: pkg:maven/org.bouncycastle/bcprov-jdk15on@1.68
17:03:10  java.lang.NullPointerException: Cannot invoke "java.net.URI.toString()" because the return value of "org.sonatype.ossindex.service.api.componentreport.ComponentReportVulnerability.getReference()" is null
17:03:10      at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform (OssIndexAnalyzer.java:350)
17:03:10      at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5 (OssIndexAnalyzer.java:256)

...

17:03:10  [WARNING] Failed to fetch component-report for: pkg:maven/com.google.guava/guava@19.0
17:03:10  java.lang.NullPointerException: Cannot invoke "java.net.URI.toString()" because the return value of "org.sonatype.ossindex.service.api.componentreport.ComponentReportVulnerability.getReference()" is null
17:03:10      at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform (OssIndexAnalyzer.java:350)
17:03:10      at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5 (OssIndexAnalyzer.java:256)

...

17:03:10  [INFO] Finished Sonatype OSS Index Analyzer (0 seconds)
17:03:10  [INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
17:03:10  [INFO] Finished Dependency Bundling Analyzer (0 seconds)
17:03:10  [INFO] Analysis Complete (4 seconds)
bjansen commented 2 years ago

Same here. OSS Index is being upgraded, and it looks like it's starting to report new vulnerabilities that are not CVEs (you need to be registered to see the details of these new types of vulnerabilities).

I tested their REST API with the following payload:

{
  "coordinates": [
    "pkg:maven/org.terracotta/offheap-store@2.5.2"
  ]
}

and got the following response:

[
  {
    "coordinates": "pkg:maven/org.terracotta/offheap-store@2.5.2",
    "description": "A library that offers data structures allocated off the java heap.",
    "reference": "https://ossindex.sonatype.org/component/pkg:maven/org.terracotta/offheap-store@2.5.2?utm_source=mozilla&utm_medium=integration&utm_content=5.0",
    "vulnerabilities": [
      {
        "id": "sonatype-2020-0267",
        "title": "1 vulnerability found",
        "description": "1 non-CVE vulnerability found. To see more details, please create a free account at https://https://ossindex.sonatype.org/ and request for this information using your registered account",
        "cvssScore": 5.9
      }
    ],
    "sonatypeOssiScore": 5.9
  }
]

It seems to be what's confusing dependency-check.

OrangeDog commented 2 years ago

@bjansen that would also explain all the false positives that have appeared at the same time. The OSS Index appears to have a bunch of new CPE/version errors.

jfurmankiewiczpros commented 2 years ago

we are getting the same

Failed to fetch component-report for: pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.12.5 java.lang.NullPointerException at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350) at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:25

ken-duck commented 2 years ago

OSS Index is now filtering "sonatype-id" vulnerability notifications for anonymous users. This may only be a partial workaround, as the "sonatype-id" vulnerability details will be visible for authenticating users.

matthewlowry commented 2 years ago

I would like to work around this temporarily by just turning off the OSS Index analyser entirely.

I thought I would be able to achieve this (Maven plugin here) by:

            <plugin>
                <groupId>org.owasp</groupId>
                <artifactId>dependency-check-maven</artifactId>
                <version>7.1.0</version>
                <configuration>
                    <!-- Turn off the OSS Index analyser. Currently causing problems with format change and rate limiting.
                        See assorted Github issues e.g. https://github.com/jeremylong/DependencyCheck/issues/4527. -->
                    <ossindexAnalyzerEnabled>false</ossindexAnalyzerEnabled>
                </configuration>
            </plugin>

But this isn't working as I expected. The analyser is still running and throwing out the errors as documented by others in this issue. What am I missing? How do I just entirely disable the OSS Index analyser if not this way?

antonilic commented 2 years ago

Hello, I am using the "org.owasp.dependencycheck" gradle plugin, version "7.1.0.1" and I am getting the same issue but for different (transitive) dependencies:

Failed to fetch component-report for: pkg:maven/xerces/xercesImpl@2.12.0
java.lang.NullPointerException
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:256)

Failed to fetch component-report for: pkg:maven/com.h2database/h2@2.1.212
java.lang.NullPointerException
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:256)

Failed to fetch component-report for: pkg:maven/org.bouncycastle/bcprov-jdk15on@1.68
java.lang.NullPointerException
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:256)

Failed to fetch component-report for: pkg:maven/javax.mail/mail@1.5.0-b01
java.lang.NullPointerException
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:256)

Failed to fetch component-report for: pkg:maven/io.netty/netty-handler@4.1.77.Final
java.lang.NullPointerException
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:256)

Failed to fetch component-report for: pkg:maven/com.google.guava/guava@31.0.1-jre
java.lang.NullPointerException
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:350)
        at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:256)

The CVEs are the following:

h2-2.1.212.jar : CVE-2018-14335
ktor-server-core-1.2.6.jar : CVE-2021-25761
ktor-server-netty-1.2.6.jar : CVE-2020-26129
spring-security-crypto-5.7.1.jar : CVE-2020-5408
xercesImpl-2.12.0.jar : CVE-2022-23437

Everything had been working ok before yesterday afternoon.

chuqa commented 2 years ago

I would like to work around this temporarily by just turning off the OSS Index analyser entirely.

I thought I would be able to achieve this (Maven plugin here) by:

            <plugin>
                <groupId>org.owasp</groupId>
                <artifactId>dependency-check-maven</artifactId>
                <version>7.1.0</version>
                <configuration>
                    <!-- Turn off the OSS Index analyser. Currently causing problems with format change and rate limiting.
                        See assorted Github issues e.g. https://github.com/jeremylong/DependencyCheck/issues/4527. -->
                    <ossindexAnalyzerEnabled>false</ossindexAnalyzerEnabled>
                </configuration>
            </plugin>

But this isn't working as I expected. The analyser is still running and throwing out the errors as documented by others in this issue. What am I missing? How do I just entirely disable the OSS Index analyser if not this way?

Using ossindexAnalyzerEnabled works for me. I use a build job, which do something like:

mvn org.owasp:dependency-check-maven:7.1.0:check -DskipProvidedScope=true -DskipRuntimeScope=true -DprettyPrint=true -DfailBuildOnCVSS=7.0 -DsuppressionFiles="..." -DossindexAnalyzerEnabled=false

No fetching issues / warnings at the moment.

davejbur commented 2 years ago

Looking at the logs it looks like dependency-check tries to access https://ossindex.sonatype.org/api/v3/component-report Viewing that in a browser gives a 405 error. That may be irrelevant, but the page does have a link on to this one: https://ossindex.sonatype.org/updates-notice which points out several breaking changes.

antonilic commented 2 years ago

@davejbur I have tried to perform a post to that link (https://ossindex.sonatype.org/api/v3/component-report) using this body (as highlighted in a comment above) 

{
  "coordinates": [
    "pkg:maven/xerces/xercesImpl@2.12.0"
  ]
}

and I receive a response describing the vulnerability stating that it isn't a CVE.

davejbur commented 2 years ago

@antonilic Sorry, I missed that:-) Thanks.

antonilic commented 2 years ago

So I don't know if these are actual CVE vulnerabilities or not... I am asking because it is breaking some builds

OrangeDog commented 2 years ago

@antonilic these are warnings about NullPointerExceptions. They don't break any builds.

antonilic commented 2 years ago

Ok thanks. In my case the build gets broken because of vulnerabilities raised by the oss index

davejbur commented 2 years ago

Indeed - the build "breaks" in that it can't be completed. As a temporary workaround, I'm using the following configuration:

      <plugin>
        <groupId>org.owasp</groupId>
        <artifactId>dependency-check-maven</artifactId>
        <version>7.1.0</version>
        <executions>
          <execution>
            <configuration>
              <ossindexAnalyzerEnabled>false</ossindexAnalyzerEnabled>
            </configuration>
            <goals>
              <goal>check</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

This compiles, skipping the remote OSS index check. Not great from a security/vulnerability checking point of view (the whole point of doing this in the first place:-)) but at least it gets past this hiccup for now.

antonilic commented 2 years ago

In gradle, I have done something like this and it works:

dependencyCheck {
    analyzers.ossIndexEnabled = false
}

I am using the plugin 7.1.0.1 for the dependency check.

However I am not sure that disabling this analyser is the right solution. It could be ok ish for now because in my case the vulnerabilities are marked as non CVE.

j-s-3 commented 2 years ago

Hey all Sonatype Product Manager for OSS Index here. Firstly sorry this has caused you all an issue. There are a number of vulnerabilities that have been found by our research teams that haven't made their way to the public databases yet. One of the huge advantages of the upgrade me made yesterday is that anyone using OSSI now benefits from research done by Sonatype's commercial research teams, making this the highest quality free data source.

These new vulnerabilities that don't have CVEs will now have SONATYPE-* IDs which I think might be causing the problem. For anonymous access we've bundled any of these new vulnerabilities up and only display the ID of the vulnerability with the highest severity. To expand them out you'll need to be authenticated.

We made a temporary fix last night to remove the aggregated vulnerability from anonymous access whenever we see a dependency-track user-agent hoping that would fix this problem. I'm guessing some of you folks are authenticated and are still running into trouble? That's probably going to require a change to dependency-track to support the new ID format.

@jeremylong

I'll reply here when I have more information.

OrangeDog commented 2 years ago

@jlstephens89 have a look at the other recent issues. Many users are now being rate-limited when they weren't before, and OSS Index has added various (apparently) new errors causing false positives.

A high-quality free public data source that wanted to improve everyone's software would not require a login in order to see it.

j-s-3 commented 2 years ago

@OrangeDog On it. https://github.com/jeremylong/DependencyCheck/issues/4535#issuecomment-1137191625

j-s-3 commented 2 years ago

@OrangeDog https://github.com/jeremylong/DependencyCheck/issues/4535#issuecomment-1137233391

philipwhiuk commented 2 years ago

Similar error, I think on an older version.

[error] org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports
[error]     at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency(OssIndexAnalyzer.java:149)
[error]     at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
[error]     at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
[error]     at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
[error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]     at java.lang.Thread.run(Thread.java:748)
[error] Caused by: java.lang.NullPointerException
[error]     at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryKey(DirectoryCache.java:149)
[error]     at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryFile(DirectoryCache.java:157)
[error]     at org.sonatype.ossindex.service.client.cache.DirectoryCache.putAll(DirectoryCache.java:134)
[error]     at org.sonatype.ossindex.service.client.internal.OssindexClientImpl.requestComponentReports(OssindexClientImpl.java:171)
[error]     at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.requestReports(OssIndexAnalyzer.java:197)
[error]     at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency(OssIndexAnalyzer.java:138)
[error]     ... 7 more
j-s-3 commented 2 years ago

Similar error, I think on an older version.

[error] org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports
[error]     at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency(OssIndexAnalyzer.java:149)
[error]     at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
[error]     at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
[error]     at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
[error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]     at java.lang.Thread.run(Thread.java:748)
[error] Caused by: java.lang.NullPointerException
[error]     at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryKey(DirectoryCache.java:149)
[error]     at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryFile(DirectoryCache.java:157)
[error]     at org.sonatype.ossindex.service.client.cache.DirectoryCache.putAll(DirectoryCache.java:134)
[error]     at org.sonatype.ossindex.service.client.internal.OssindexClientImpl.requestComponentReports(OssindexClientImpl.java:171)
[error]     at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.requestReports(OssIndexAnalyzer.java:197)
[error]     at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency(OssIndexAnalyzer.java:138)
[error]     ... 7 more

Hmm from that stack trace it looks like the the null coordinates has been cached in some way. Looking at the OSSI client code this is getting cached in your user directory. We've just released more bug fixes which should help so try again and if that still fails, try clearing the cache.

image

DiegoMellisG commented 2 years ago

Same problem here but not with maven, is with flask

[WARN] Failed to fetch component-report for: pkg:pypi/flask@2.1.1 java.lang.NullPointerException: null at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform(OssIndexAnalyzer.java:325) at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5(OssIndexAnalyzer.java:231) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:4[72](https://gitlab.falabella.com/falabella-retail/txd/IA/automatizaciones/bot-mkp/mkp-get-business-report/-/jobs/6866345#L72)) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:1[73](https://gitlab.falabella.com/falabella-retail/txd/IA/automatizaciones/bot-mkp/mkp-get-business-report/-/jobs/6866345#L73)) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:490) at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.enrich(OssIndexAnalyzer.java:232) at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency(OssIndexAnalyzer.java:156) at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131) at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88) at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:[74](https://gitlab.falabella.com/falabella-retail/txd/IA/automatizaciones/bot-mkp/mkp-get-business-report/-/jobs/6866345#L74)8)

uwesinha commented 2 years ago

Currently, I keep getting NPEs for these libraries (dependency-check-maven plugin 7.1.0): 

 pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.12.4
 pkg:maven/org.apache.commons/commons-dbcp2@2.8.0
 pkg:maven/org.bouncycastle/bcprov-jdk15on@1.69
 pkg:maven/com.squareup.okhttp/okhttp@2.7.5
 pkg:maven/org.apache.xmlgraphics/batik-svgbrowser@1.13
 pkg:maven/com.fasterxml.woodstox/woodstox-core@5.2.1
 pkg:maven/com.twelvemonkeys.imageio/imageio-jpeg@3.3.2
 pkg:maven/org.terracotta/offheap-store@2.5.2

As suggested by the others, setting <ossindexAnalyzerEnabled>false</ossindexAnalyzerEnabled> in the POM helped me get rid of these NPEs. But of course, I'd rather have my project dependencies checked against OSS Index again...

Addendum 1: Stacktrace is always the same:

[WARNING] Failed to fetch component-report for: pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.12.4
java.lang.NullPointerException
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.transform (OssIndexAnalyzer.java:350)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.lambda$enrich$5 (OssIndexAnalyzer.java:256)
    at java.util.stream.ReferencePipeline$3$1.accept (ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining (ArrayList.java:1384)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential (ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential (ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEachOrdered (ReferencePipeline.java:490)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.enrich (OssIndexAnalyzer.java:257)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency (OssIndexAnalyzer.java:163)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:131)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)

Addendum 2: Weirdly enough, the Jenkins plugin doesn't seem to be affected.

j-s-3 commented 2 years ago

@uwesinha have you tried in the last 20 mins? I've passed your message onto the team and we will verify those specific packages.

uwesinha commented 2 years ago

Thanks for looking into this, @jlstephens89! I ran my build again 8 minutes ago, but sadly no change. 😢 (I'm in CEST land, so I'll check back tomorrow. 😃)

antonilic commented 2 years ago

Thank you, i have run the build with gradle and I am not seeing the null pointer exc anymore. I see that there are some vulnerabilities that should be solved our side.

ken-duck commented 2 years ago

@uwesinha I made a small project with your packages, but was not able to reproduce the exception. I wonder if the data in the dependency-check cache is corrupted. I would suggest deleting the OSS Index portion of the cache and rerunning.

I found it in the <dependency-check install dir>/data/oss_cache folder. Clear that out and see if it resolves the problem.

somera commented 2 years ago

I don't see the NullPointerException anymore. Thx!

ken-duck commented 2 years ago

@uwesinha Another edge case has been resolved which may also have been the cause for your NPE.

davejbur commented 2 years ago

Thanks all - I've just retried with ossindexAnalyzerEnabled set to true again, and it all works fine!

j-s-3 commented 2 years ago

We at Sonatype really appreciate everyone's patience as we work through these issues. I can assure you we did extensively test these changes over the last 2 months but there is always room for improvement and the next change we make we'll be doing a lot more testing with the various clients and looking for ways to more gradually roll out. You all reporting and raising the issues has enabled us to react as fast as possible and you've all been a great help. Thanks!

Also if you ever need to reach us directly you can at ossindex@sonatype.org

philipwhiuk commented 2 years ago

Thanks for the service in the first place!

As client users we should probably give the maintainer a hand in making the library a bit more resilient to unexpected formatted data so that the tool handles it more gracefully.

I’ll try to find some time to help with that over the coming week.

uwesinha commented 2 years ago

Thanks erveryone, esp. @ken-duck, for analyzing this problem! After deleting the OSS-Index-related portion of the cache the NPE is gone at last.

BTW: For the Maven plugin (version 7.x) the cache directories can be found under $HOME/.m2/repository/org/owasp/dependency-check-data/7.0.

alshopov commented 2 years ago

Alteranively one can issue the command

mvn dependency-check:purge

https://jeremylong.github.io/DependencyCheck/dependency-check-maven/purge-mojo.html

aikebah commented 2 years ago

@jeremylong @jlstephens89 Think we can close this now? Or are there still reports in the comments that require follow-up?

OrangeDog commented 2 years ago

@aikebah as far as I understand, dependency-check still needs updating to handle the change. Then OSSIndex can stop hiding it from dependency-check user agents.

aikebah commented 2 years ago

@aikebah as far as I understand, dependency-check still needs updating to handle the change. Then OSSIndex can stop hiding it from dependency-check user agents.

Ah right... after carefully reading the history I see that sonatype currently has a quickfix in place to hide the problematic entries for dependencycheck

aikebah commented 2 years ago

A local test with a modified current snapshot (modified user-agent, so that useragent based response filtering at Sonatype OSSIndex will not kick in) appears to suggest that the issue is resolved.

I got proper reports for both anonymous access and authenticated access for pkg:maven/io.netty/netty-handler@4.1.77.Final as well as pkg:maven/com.fasterxml.jackson.core/jackson-core

Unauthenticated yields a report with a 1 vulnerability found (OSSINDEX)

Authenticated (after clearing cache) yields the detailed sonatype vulnerability

@jlstephens89 @ken-duck can you confirm that https://github.com/sonatype/ossindex-public/issues/35#issuecomment-1141267867 essentially means that this issue has been resolved by internal fixes in the OSSIndex? Or did I overlook some cases?

The only part I wasn't able to test is a result of multiple sonatype internal vulnerability records for a single component. The packageURLs that I checked from several of the NPE reports all appear to have only a single sonatype internal vulnerability record.

ken-duck commented 2 years ago

@aikebah It is definitely my understanding that the comment you linked means that the issue has been resolved by internal fixes. The fundamental problem was that the "reference" was not being set in some cases and that wreaked havoc on several downstream tools. The quick fix hid these things, but a proper fix has been applied.

In addition (and off topic), a few recent fixes have also improved the filling in of the "cve" field when a vulnerability has a sonatype-#### Id, but subsequently is assigned a CVE number. In those cases the "id" will remain as a sonatype-#### id, but the cve field will be appropriately filled in.

In either case the reference should be assigned.

If we get enough community feedback that supports no longer filtering the "sonatype-####" from the unauthenticated results we would certainly be willing and happy to do so.

ThomGeG commented 2 years ago

If we get enough community feedback that supports no longer filtering the "sonatype-####" from the unauthenticated results we would certainly be willing and happy to do so.

@ken-duck so does that mean that it's intentional that unauthenticated DependencyCheck requests aren't currently returning all the same vulnerabilities an authenticated or non-DependencyCheck request does?

I recently raised an issue, https://github.com/sonatype/ossindex-public/issues/37, about a discrepancy I've been seeing between what's being returned from OSS based on your user-agent. It's causing my Jenkins build server (which authenticates for better rate limiting) to have a different outcome than my local scans (which doesn't bother authenticating) that's taken a few days to lead me here.

It seems it would be better to have the full list of vulnerabilities returned regardless of authentication now that DependencyCheck has fixed the issue, no? Otherwise there would be what I imagine is a large group of people potentially missing vulnerabilities due to never authenticating as rate limiting isn't a concern for them.

aikebah commented 1 year ago

If we get enough community feedback that supports no longer filtering the "sonatype-####" from the unauthenticated results we would certainly be willing and happy to do so.

@ken-duck We are now getting false negative reports that originate from our users due to the filtering. I think it would be fine to remove the filtering by now. Have never encountered any issue using it in authenticated mode.