search

stevespringett / nist-data-mirror

A simple Java command-line utility to mirror the CVE JSON data from NIST.
Apache License 2.0
206 stars 93 forks source link

Recommended way to deal with failed downloads? #39

Open ghost opened 4 years ago

ghost commented 4 years ago

In the light of issues like #38: What is the recommended way to use this tool when downloads can fail?

Is it safe to use this tool without risking to corrupt an existing mirror directory? If I have used the tool successfully to mirror the JSON files into a directory nist/, will this directory still contain the data in a usable way even if running this tool again fails?

Edit: If I look at the code

https://github.com/stevespringett/nist-data-mirror/blob/5e5ef4ad779d74820bdd250d0dc00c691128e4e2/src/main/java/us/springett/nistdatamirror/NistDataMirror.java#L196

the target file is directly opened for writing. This can potentially corrupt it. Maybe a temporary file could be used instead?

ghost commented 4 years ago

I've opened a PR #45 that implements downloading to a temporary directory.

nigredo-tori commented 4 years ago

+1 to this issue. I've stumbled upon this project when issues with nvd.nist.gov broke CI in our projects. I intended to use the mirror to avoid this issue in the future. However, if connection issues can, indeed, break the mirror, it defeats the point.

lbreuss commented 1 year ago

+1 We suffer the same issue with 1.5.3, and now upgraded to docker image nvd-mirror 1.6.0. But I expect the problem to show again in a few days. This is quite a problem for our CI system, as the maven dependency-check plugin caches the files itself but does not handle corrupt .json.gz files very well, i.e. it does not try to immediately fetch again when I've already fixed the nvd-mirror manually...

IMHO, nist-data-mirror should quarantine downloaded corrupt files. Or -- as suggested by @ghost -- download to a temporary file, before moving the integrity-checked file to the output directory, i.e. /tmp/nvd.