search

stevespringett / nist-data-mirror

A simple Java command-line utility to mirror the CVE JSON data from NIST.
Apache License 2.0
206 stars 93 forks source link

java.io.EOFException: Unexpected end of ZLIB input stream #78

Closed daxin09pp closed 2 years ago

daxin09pp commented 3 years ago

hi ,

How can I solve this? I use the latest version of NVD-Data-Mirror.

C:\Users\Administrator\Downloads>java -Djdk.http.auth.tunneling.disabledSchemes=
"" -Djdk.http.auth.proxying.disabledSchemes="" -Dhttps.proxyHost=10.1.54.181 -Dh
ttps.proxyPort=1080 -jar nist-data-mirror.jar C:\nvd\
Downloading files at Mon Feb 22 16:41:35 ULAT 2021
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz
Download succeeded nvdcve-1.1-modified.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.util.zip.InflaterInputStream.fill(Unknown Source)
        at java.util.zip.InflaterInputStream.read(Unknown Source)
        at java.util.zip.GZIPInputStream.read(Unknown Source)
        at java.io.FilterInputStream.read(Unknown Source)
        at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.
java:232)
        at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.
java:218)
        at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java
:139)
        at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:8
4)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta

Thanks!

daxin09pp commented 3 years ago

other Linux Server without proxy also not work..

XXXX@XXX-25-201:~/nvd-mirror$ java -jar /home/XXXX/nvd-mirror/nist-data-mirror2.jar /home/XXXX/nvd-mirror/data/
Downloading files at Mon Feb 22 17:03:56 CST 2021
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta
Download succeeded nvdcve-1.1-2002.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.meta
Download succeeded nvdcve-1.1-2003.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.meta
Download succeeded nvdcve-1.1-2004.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.json.gz
Download succeeded nvdcve-1.1-2004.json.gz
Uncompressed nvdcve-1.1-2004.json
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.meta
Download succeeded nvdcve-1.1-2005.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.json.gz
Download succeeded nvdcve-1.1-2005.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
        at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:232)
        at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:218)
        at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:160)
        at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:142)
        at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:84)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.meta
Download succeeded nvdcve-1.1-2006.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz
Download succeeded nvdcve-1.1-2006.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
        at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:232)
        at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:218)
        at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:160)
        at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:142)
        at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:84)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.meta
FavorMylikes commented 3 years ago

Same issue, I think we need to check the original file form nist database.

sellersj commented 3 years ago

Even though it's downloading httpS files, try to use the http (no S) proxy config. That is what has worked for me in the past.

Jim

On Tue, Apr 13, 2021 at 8:12 AM FavorMylikes @.***> wrote:

Same issue, I think we need to check the original file form nist database.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/stevespringett/nist-data-mirror/issues/78#issuecomment-818686238, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAABCJQWQR3LX5Z43R5NEGDTIQYKVANCNFSM4YADEVJQ .

FavorMylikes commented 3 years ago

@daxin09pp After compare the gz file form https://nvd.nist.gov and local, I can be sure that some error bytes be inserted into the correct file.

It's kind of little weird, but I'm not sure how this happened.

Maybe it's a network problem.

So I use proxy by adding -Dhttp.proxyHost=YOUR_HTTP_PROXY_HOST -Dhttp.proxyPort=YOUT_HTTP_PROXY_PORT to redownload from https://nvd.nist.gov

That works again

daxin09pp commented 3 years ago

@FavorMylikes @sellersj
Thanks guys. It's really a network issue. When I use the proxy or the network is better, it is not a issue.

PascalTurbo commented 2 years ago

We are also running into this issue from time to time. NVD provides the Size and the SHA256 Hash with the meta information. Would it be possible to add at least a check of the gz file size. At best uncompress the gz and compare the hash of the json file. If there is a different, download the file again.

PascalTurbo commented 2 years ago

@stevespringett would be great to merge the validCheck implemented by @falco9 as it should solve our problem with invalid files coming up once a week.

stevespringett commented 2 years ago

Fix has been merged. Thanks @falco9 for the PR.