search

stevespringett / nist-data-mirror

A simple Java command-line utility to mirror the CVE JSON data from NIST.
Apache License 2.0
206 stars 93 forks source link

nvdcve-1.1-2013.json.gz corrupt with nist-data-mirror 1.6.0 #145

Closed rmontag-ap closed 1 year ago

rmontag-ap commented 1 year ago

Hello, we are currently facing a weird issue with our nist-data-mirror for approx. a week. The file nvdcve-1.12013.json.gz gets corrupt and cannot be extracted:

$ gunzip nvdcve-1.1-2013.json.gz

gzip: nvdcve-1.1-2013.json.gz: unexpected end of file

When the file is corrupt and we run nist-data-mirror 1.6.0 on our data folder we are getting the following output:

[Thu Oct  6 09:16:08 CEST 2022]
Downloading files at Thu Oct 06 09:16:08 CEST 2022
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.meta
Download succeeded nvdcve-1.1-recent.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta
Download succeeded nvdcve-1.1-2002.meta
File 2002 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.meta
Download succeeded nvdcve-1.1-2003.meta
File 2003 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.meta
Download succeeded nvdcve-1.1-2004.meta
File 2004 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.meta
Download succeeded nvdcve-1.1-2005.meta
File 2005 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.meta
Download succeeded nvdcve-1.1-2006.meta
File 2006 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.meta
Download succeeded nvdcve-1.1-2007.meta
File 2007 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2008.meta
Download succeeded nvdcve-1.1-2008.meta
File 2008 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2009.meta
Download succeeded nvdcve-1.1-2009.meta
File 2009 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2010.meta
Download succeeded nvdcve-1.1-2010.meta
File 2010 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.meta
Download succeeded nvdcve-1.1-2011.meta
File 2011 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.meta
Download succeeded nvdcve-1.1-2012.meta
File 2012 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
File 2013 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
The File 2013 is corrupted
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.meta
Download succeeded nvdcve-1.1-2014.meta
File 2014 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.meta
Download succeeded nvdcve-1.1-2015.meta
File 2015 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2016.meta
Download succeeded nvdcve-1.1-2016.meta
File 2016 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.meta
Download succeeded nvdcve-1.1-2017.meta
File 2017 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2018.meta
Download succeeded nvdcve-1.1-2018.meta
File 2018 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.meta
Download succeeded nvdcve-1.1-2019.meta
File 2019 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.meta
Download succeeded nvdcve-1.1-2020.meta
File 2020 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2021.meta
Download succeeded nvdcve-1.1-2021.meta
File 2021 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2022.meta
Download succeeded nvdcve-1.1-2022.meta
File 2022 is valid.

So it reports that the file is corrupted; but I have no idea what has caused the corruption and how to solve it. We have a cronjob that is running the nist-data-mirror every hour.

I already deleted all nvdcve-1.1-2013* files and rerun the nist-data-mirror:

...
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.meta
Download succeeded nvdcve-1.1-2011.meta
File 2011 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.meta
Download succeeded nvdcve-1.1-2012.meta
File 2012 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz
Download succeeded nvdcve-1.1-2013.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
        at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:263)
        at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:249)
        at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:191)
        at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:155)
        at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:87)
File 2013 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
The File 2013 is corrupted
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.meta
Download succeeded nvdcve-1.1-2014.meta
File 2014 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.meta
Download succeeded nvdcve-1.1-2015.meta
File 2015 is valid.
...

So the download fron the nist-data-mirror failed. I did a manual download from the same server and it worked without problem:

$ wget https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz
--2022-10-06 09:33:35--  https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz
Resolving nvd.nist.gov (nvd.nist.gov)... 18.235.227.114, 2600:1f18:268d:1d01:f609:5e91:8a48:f546
Connecting to nvd.nist.gov (nvd.nist.gov)|18.235.227.114|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2449748 (2.3M) [application/x-gzip]
Saving to: ‘nvdcve-1.1-2013.json.gz’

100%[=======================================================================================================================>] 2,449,748   2.38MB/s   in 1.0s

2022-10-06 09:33:37 (2.38 MB/s) - ‘nvdcve-1.1-2013.json.gz’ saved [2449748/2449748]

Any advice and help would be appreciated.

Greetings, Rainer

rmontag-ap commented 1 year ago

My current workaround is indeed to 1) run nist-data-mirror 2) remove offending file nvdcve-1.1-2013.json.gz 3) manual download "wget https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz"

rmontag-ap commented 1 year ago

Additional info: We are using nist-data-mirror 1.6.0 since June 2022, but this issue has started last week and has hit us now three times in a row in the last days.

rmontag-ap commented 1 year ago

I did some further checks and the checksum on the unzipped 2013 is not matching:

$ cat nvdcve-1.1-2013.meta
lastModifiedDate:2022-09-30T03:01:57-04:00
size:45431615
zipSize:2449884
gzSize:2449748
sha256:4DF6DAF5270EEA9F79D316297EBCC70352A0BFF40F49A8715A7E6C621B55CBAA
$ sha256sum nvdcve-1.1-2013.json
95a2e870cc5865c11fcc4b63e98d633ca249899011244f74bcd2254127e39f62  nvdcve-1.1-2013.json
$ stat -c %s nvdcve-1.1-2013.json.gz
2449748

So I now extracted the manual downloaded file and now the sha256 matches and nist-data-mirror is not complaining about 2013:

...
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.meta
Download succeeded nvdcve-1.1-2012.meta
File 2012 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
File 2013 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.meta
Download succeeded nvdcve-1.1-2014.meta
File 2014 is valid.
...

I still need to find out how both the nvdcve-1.1-2013.json.gz and the nvdcve-1.1-2013.json got corrupted on my system.

twwd commented 1 year ago

We suffer the same issue with nist-data-mirror 1.5.0 but with the nvdcve files for first 2020 and then 2014. A check that only valid files are replaced would be great.

rmontag-ap commented 1 year ago

After deleting both (extracted) json and json.gz file, the nist-data-mirror is working as expected. I did not find out, what has caused the corruption of the 2013 file.

lbreuss commented 1 year ago

We suffer the same issue with 1.5.3, and now upgraded to docker image nvd-mirror 1.6.0. But I expect the problem to show again in a few days. This is quite a problem for our CI system, as the maven dependency-check plugin caches the files itself but does not handle corrupt .json.gz files very well, i.e. it does not try to fetch again when I've already fixed the nvd-mirror manually...

I don't think the issue is solved. I moved my other comments to #39

rmontag-ap commented 1 year ago

I agree, I closed my issue, but the core problem is not solved. As we are doing the distribution of the downloaded files with another tool, we changed our distribution tool to check the integrity of the cve files (again) and block the upload in case of a file corruption.