search

DependencyTrack / dependency-track

Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain.
https://dependencytrack.org/
Apache License 2.0
2.61k stars 553 forks source link

NVD full download again #3328

Closed somera closed 3 weeks ago

somera commented 9 months ago

Current Behavior

I run my DT 4.10.0 in docker. NVD will be synced every 6 days (cause I don't have any projects at the moment). Today I added my NVD key cause I forgot do this. But now I see, that the Last Modification is not set

image

image

And as I see today is the 4th time that DT is syncing

image

Steps to Reproduce

Just try the same what I#m doing.

Expected Behavior

Sync only deltas.

Dependency-Track Version

4.10.0

Dependency-Track Distribution

Container Image

Database Server

PostgreSQL

Database Server Version

No response

Browser

Google Chrome

Checklist

nscuro commented 9 months ago

This was fixed in https://github.com/DependencyTrack/dependency-track/pull/3322 and has already been released with 4.10.1: https://docs.dependencytrack.org/changelog/

However it looks like your instance is not using the NVD REST API (the NvdParser class appearing in your logs is only invoked by the legacy feed mirroring, which does not support incremental updates).

somera commented 9 months ago

This was fixed in #3322 and has already been released with 4.10.1: https://docs.dependencytrack.org/changelog/

I saw the new version. I will update after the current run.

However it looks like your instance is not using the NVD REST API (the NvdParser class appearing in your logs is only invoked by the legacy feed mirroring, which does not support incremental updates).

This

image

is my config. I added today only my API key.

And it's not possible to see the API key after I set it?

freddiN commented 9 months ago

I'm not convinced that this new API based process is actually working: I just updated to 4.10.1 and saw this after the restart:

2023-12-19 12:09:39,478 INFO [PortfolioMetricsUpdateTask] Completed portfolio metrics update in 00:06:832
2023-12-19 12:10:22,647 INFO [VulnDbSyncTask] Starting VulnDB mirror synchronization task
2023-12-19 12:10:22,647 INFO [VulnDbSyncTask] VulnDB mirror directory does not exist. Skipping.
2023-12-19 12:10:22,672 INFO [NistApiMirrorTask] CVEs were not previously mirrored via NVD API; Will mirror all CVEs
2023-12-19 12:10:24,428 ERROR [NistApiMirrorTask] An unexpected error occurred while mirroring the contents of the National Vulnerability Database
io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 404
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:352)
        at org.dependencytrack.tasks.NistApiMirrorTask.inform(NistApiMirrorTask.java:166)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2023-12-19 12:10:24,428 INFO [NistApiMirrorTask] Mirroring of 0 CVEs completed in PT1.758116876S

Configured like the pic in the post before this one, my prsonal NVD key is configured.

I'm also using the docker variant, but 4.10.1 now.

nscuro commented 9 months ago

Yeah, unfortunately the REST API has been problematic stability-wise. Like Dependency-Check, we rely on https://github.com/jeremylong/Open-Vulnerability-Project to communicate with the API. The author of that library has gone to great lengths for dealing with API oddities and availability issues, but one can't simply retry on non-retryable errors like HTTP 404s...

FWIW, the API returns a 404 when the API key is invalid / the NVD thinks it doesn't exist. There have been occasions where previously working keys "broke", but I don't know why.

Keep in mind you can switch back to the previous, file-based mirroring for the time being: https://groups.google.com/a/list.nist.gov/g/nvd-news/c/aofnAd3HP2g

somera commented 9 months ago

After update to 4.10.1 image

I'm stopping this after 1 hour with all the warnings.

nscuro commented 9 months ago

Is this an bug?

The NVD API is indicating a transient error and the client is retrying.

Should I run it?

Run what?

What happens there?

The NVD API is indicating a transient error and the client is retrying.

somera commented 9 months ago

Should I run it?

Run what?

DT. For 1 hour I saw 10% CPU usage on my system and all the warnings. When I see this than I'm stopping the tool.

When "NVD API is indicating a transient error and the client is retrying" -> than DT should not try this in a loop. If there is a problem, than try this in 30 or 60 minutes again. Here is an better retry pattern needed. What if the problem exists for 2 weeks?

somera commented 9 months ago

I don't see any issues with the dependency-check-maven plugin 9.0.7 which can download the delta.

[INFO] Checking for updates
[INFO] NVD API has 580 records in this update
[INFO] Downloaded 580/580 (100%)
[INFO] Completed processing batch 1/1 (100%) in 828ms
valentijnscholten commented 9 months ago

I don't see any issues with the dependency-check-maven plugin 9.0.7 which can download the delta.

DT and DC both use the same library to sync the NVD API.

somera commented 9 months ago

ok, not I see the date. Needed 3-4 restarts.

image

somera commented 9 months ago

The NVD API is indicating a transient error and the client is retrying.

How long will DT retry this? Cause I see this again.

It's possible to get more details, what happens here? I set the LOGGING_LEVEL

- LOGGING_LEVEL=TRACE

or

- LOGGING_LEVEL=DEBUG

But I coudn't see more log informations.

After one hour image I stopped DT.

somera commented 9 months ago

For over 1 hour I saw the retry warning in the log. As I wrote there is other retry pattern needed. I setet the NVD API again. I restarted it and than the sync worked.

For me it looks like DT is unstable.

valentijnscholten commented 9 months ago

There's exponential back-off and support for the Retry-After http response header. Not sure why the timestamps in your logs seem to indicate something else.

https://github.com/DependencyTrack/dependency-track/blob/3eea6989a60440cf031fbecd180afa3565042ea1/src/main/java/org/dependencytrack/tasks/NistApiMirrorTask.java#L507-L528

The logs indicate the sync is progressing, despite the NVD API being unstable. The sync can take a couple of hours to complete. If you have an API Key you should use it, much less chances of errors (and retries).

Personally I have stopped trying to get it to work, the NVD API is too unstable at present. Chances are NVD will reinstate some kind of bulk feeds for bulk synchronization.

somera commented 9 months ago

There's exponential back-off and support for the Retry-After http response header. Not sure why the timestamps in your logs seem to indicate something else.

That's what I'm observing.

The logs indicate the sync is progressing, despite the NVD API being unstable. The sync can take a couple of hours to complete. If you have an API Key you should use it, much less chances of errors (and retries).

But it will be better to show what is the error response from NVD API. It helps to understand the problem. And I'm using the API Key. Cause I can see it in the UI I'm not sure this is used. This is my hope.

Personally I have stopped trying to get it to work, the NVD API is too unstable at present. Chances are NVD will reinstate some kind of bulk feeds for bulk synchronization.

Thx for the info.

somera commented 3 weeks ago

I can close it.