Scan failing after 6 hours

lread commented 3 months ago

Description

I use nvd-clojure on CI on a clj-yaml and pomegranate. Since June 28th, these nvd scans are failing after 6h or so. In nvd-clojure logs I see plenty of:

2024-07-01 00:32:49.995] INFO AsyncHttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.RequestNotExecutedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
[2024-07-01 00:34:08.750] WARN NvdApiRetryStrategy - NVD API request failures are occurring; retrying request for the 5 time
[2024-07-01 00:35:03.023] INFO AsyncHttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.RequestNotExecutedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
[2024-07-01 00:35:56.519] WARN NvdApiRetryStrategy - NVD API request failures are occurring; retrying request for the 5 time
[2024-07-01 00:37:29.659] WARN NvdApiRetryStrategy - NVD API request failures are occurring; retrying request for the 5 time
[2024-07-01 00:38:10.802] WARN NvdApiRetryStrategy - NVD API request failures are occurring; retrying request for the 6 time
[2024-07-01 00:41:09.156] INFO AsyncHttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.RequestNotExecutedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
[2024-07-01 00:42:38.186] INFO AsyncHttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.ConnectionClosedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
[2024-07-01 00:44:21.237] INFO AsyncHttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.ConnectionClosedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
[2024-07-01 00:45:00.987] WARN NvdApiRetryStrategy - NVD API request failures are occurring; retrying request for the 5 time
[2024-07-01 00:48:47.817] INFO AsyncHttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.RequestNotExecutedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
[2024-07-01 00:49:31.932] WARN NvdApiRetryStrategy - NVD API request failures are occurring; retrying request for the 5 time

It seems that many folks are reporting connectivity issues at DependencyCheck. I'm not sure, but it might be that NIST data feeds are misbehaving these days.

I experimented on clj-yaml by bumping org.owasp/dependency-check-core to version 10.0.0. And it might have helped. Then again I might have just gotten lucky.

But... It probably would not hurt to bump dependency-check to 10.0.0 in nvd-clojure.

Version

4.0.0

Java version

Was running:
openjdk version "11.0.23" 2024-04-16
OpenJDK Runtime Environment Temurin-11.0.23+9 (build 11.0.23+9)
OpenJDK 64-Bit Server VM Temurin-11.0.23+9 (build 11.0.23+9, mixed mode)

Switch to jdk21 when I bumped DependencyCheck
openjdk 21.0.3 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

Installation compliance

[X] I have read again and made sure that I'm following exactly the instructions for my tool of choice (Leiningen, Clojure CLI, Clojure CLI Tool).
[X] I understand that false positives can be skipped locally and should be reported to DependencyCheck.

vemv commented 3 months ago

Hi Lee!

Thanks for the report.

It would be reasonable to infer that if a web API is down, then it's down for all consumers.

I'll leave the issue open for a while for visibility.

But yes we should bump the core dependencies, regardless. Not good timing if my attempt would fail anyway due to the API downtime.

Cheers - V

lread commented 3 months ago

Thanks for the reply @vemv!

There seems to be more than one thing going on:

NIST data feeds are having issues
v9.0.8 of dependency-check has no success in downloading the db (at least for me) it fails after 6h on CI

I tried bumping dependency-check to 10.0.0 in pomegranate and clj-yaml and observed what happened on GitHub Actions CI. Running a scan when no db is already cached is slow, but I have not seen it fail at the task. Example run times: 37m, 23m, 31m, 33m (back when things were working better, 12m to 15m seemed normal to download the entire db).

So, it seems that bumping to 10.0.0 helps.

That said... I am seeing quite a few lines like this in the logs:

[2024-07-02 03:51:36.108] ERROR CveDB - Updating CVE: CVE-2024-5635
[2024-07-02 03:51:36.109] ERROR CveDB - Updating CVE: CVE-2024-5636
[2024-07-02 03:51:36.110] ERROR CveDB - Updating CVE: CVE-2024-5262
[2024-07-02 03:51:36.122] ERROR CveDB - Updating CVE: CVE-2024-5184

I assume that the scan download work recovers from these errors because the overall operation does not fail, but I could be wrong.

(I also adjusted our CI jobs to actually properly cache the db. We were being NIST data feed rude and effectively not caching past 1 day.)

lread commented 3 months ago

Ah, the example of ERROR log lines I showed above was an accident and can be ignored and is fixed in 10.0.1:

https://github.com/jeremylong/DependencyCheck/issues/6746#issuecomment-2202578571

https://github.com/jeremylong/DependencyCheck/commit/8c731cd63f6323a2be072c31530df0284e387f4d

lread commented 3 months ago

@vemv update from the field: dependency-check 10.0.1 seems to be working well enough. Full db download can be very slow, but unlike the current 9.0.8 dep, 10.0.1 works.

vemv commented 3 months ago

Thanks! Let's see if I can update the core dep asap.

Please nudge me if I don't.

svdo commented 3 months ago

I can confirm that I had this same issue and overriding the dependency-check dep to 10.0.1 also fixes it for me.

lread commented 3 months ago

10.0.2 is out, and apparently, it is a mandatory update.

lread commented 3 months ago

I just tried 10.0.2 with clj-yaml locally and in https://github.com/clj-commons/clj-yaml/pull/129. It looks much better in that I don't see retries during the db download in the logs. And it completed the scan in a snappy(?) 18m (local) 15m (PR up on CI). This speed is close enough to what it used to be for a full db download before the slowdowns.

vemv commented 3 months ago

Thanks for the notes!

I'd recommend to override the dep as I can't guarantee that I'll be able to cut a release soon.

Most times it's as easy as that.

(there are other unrelated TODOs which is why I'm delaying a release)

lread commented 2 months ago

Updates from the field: I've updated to dependency-check to 10.0.3 for clj-yaml and pomegranate. Scans where the entire nvd DB is downloaded (which now only happens for me when bumping dependency-check) were much faster and took about ~3min on CI.

falcowinkler commented 1 month ago

how do you bump the dependency check locally? i have added the override to project.clj of the helper project

(defproject nvd_check_helper_project "local"
  :description "A helper project to assist Ardoq API with its vulnerability scanning"
  :dependencies [[nvd-clojure "4.0.0"]
                 [org.owasp/dependency-check-core "10.0.3"]
                 [org.clojure/clojure "1.11.1"]]
  :jvm-opts ["-Dclojure.main.report=stderr"])

But now i am getting java.lang.ClassNotFoundException org.slf4j.helpers.LegacyAbstractLogger

Anyone have an example of fully working setup?

vemv commented 1 month ago

You can view dependency-check-core's dependencies in:

https://mvnrepository.com/artifact/org.owasp/dependency-check-core/10.0.3

You can declare them explicitly in that project.clj or exclude conflicting dependencies using :exclusions:

[nvd-clojure "4.0.0" :exclusions [,,,]]

(do this only for log4j stuff)

This is a quite common thing to do in Clojure projects. Anyway feel free to share the result once you get it working.

lein deps :tree helps as well.

(and apologies for my limited availability - I've been sprinting at work for a long while)

solita-antti-mottonen commented 1 month ago

@falcowinkler This worked for me:

(defproject nvd-helper "local"
            :description "nvd-clojure helper project"
            :dependencies [[nvd-clojure "4.0.0"
                            ;; Replaced by a newer version until NVD-Clojure is updated
                            :exclusions [org.owasp/dependency-check-core]]
                           [org.owasp/dependency-check-core "10.0.3"]
                           ;; Dependency-check-core brings in older version which doesn't work
                           [org.slf4j/slf4j-api "2.0.10"]
                           [org.clojure/clojure "1.11.3"]]
            :jvm-opts ["-Dclojure.main.report=stderr"])

danielcompton commented 1 month ago

If we're installing nvd-clojure as a Clojure Tool, what is the right method to override the dependency-check-core and slf4j dependencies?

vemv commented 1 month ago

Apparently you can:

clojure -Ttools install nvd-clojure/nvd-clojure '{:mvn/version "RELEASE"}' :as nvd
clj -Sdeps '{:aliases {:change {:override-deps {clansi/clansi {:mvn/version "1.0.0"}}}}}' -A:change -Tnvd

clansi/clansi is an example, the actual overrides are above

Source

falcowinkler commented 1 month ago

Thanks @solita-antti-mottonen, the setup works perfectly now.

rm-hull / nvd-clojure