search

sjvermeu / cvechecker

Command-line utility to scan the system and report on potential vulnerabilities, based on public CVE data
GNU General Public License v3.0
258 stars 68 forks source link

pullcves shows useless errors when proxy settings are missing #39

Open wjcgy opened 5 years ago

wjcgy commented 5 years ago

[root@localhost cvechecker-master]# cvechecker -i [root@localhost cvechecker-master]# pullcves pull Converting nvdcve-2.0-2002.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2002.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2002.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2002.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2003.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2003.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2003.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2003.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2004.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2004.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2004.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2004.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2005.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2005.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2005.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2005.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2006.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2006.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2006.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2006.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2007.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2007.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2007.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2007.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2008.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2008.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2008.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2008.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2009.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2009.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2009.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2009.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2010.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2010.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2010.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2010.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2011.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2011.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2011.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2011.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2012.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2012.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2012.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2012.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2013.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2013.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2013.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2013.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2014.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2014.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2014.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2014.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2015.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2015.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2015.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2015.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2016.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2016.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2016.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2016.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2017.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2017.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2017.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2017.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2018.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2018.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2018.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2018.csv into database 0 records processed (0 already in db)... Converting nvdcve-2.0-2019.xml to CSV... gzip: /usr/local/var/cvechecker/cache/nvdcve-2.0-2019.xml.gz: unexpected end of file -:1: parser error : Document is empty unable to parse - ok Loading in nvdcve-2.0-2019.csv in cvechecker. Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-2.0-2019.csv into database 0 records processed (0 already in db)... Downloading nvdcve-2.0-Modified.xml... ok (not downloaded, same file) Downloading versions.dat... ok (not downloaded, same file)

airbjorn commented 5 years ago

I'm experiencing the same behavior with cvechecker v3.8 (@ CentOS7). So maybe behavior is due to a mismatched remote database..?

airbjorn commented 5 years ago

I found out that from within our company network, http://nvd.nist.gov/ is not reachable. This might be due to the proxy configuration. @wjcgy : is it also within a company's network you try that in?

Because: from that site the download is wget'ed from. pullcves is just a script you can examine easily.

However btw, on my mobile the site works at the same time, via my mobile provider.

sjvermeu commented 5 years ago

The pullcves script is indeed quite simple. Perhaps too simple, as it doesn't do proper exit validation on the wget command :-(

The command itself is something like the following:

~$ wget -q -O nvdcve-2.0-2019.xml.gz https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2019.xml.gz

If that command fails, drop the "-q" to get more output on why it fails. If it is indeed because you don't have an open Internet connection, you can force the script to use a different (updated) command that for instance point to the proxy:

~$ export WGETCMD="wget -e use_proxy=on -e http_proxy=:" ~$ pullcves pull

If you are allowed to have the proxy settings system-wide, you can just update /etc/wgetrc (or in ~/.wgetrc if you have a dedicated user for cvechecker).

I'll definitely need to add in some checking in this script.

airbjorn commented 5 years ago

In my case, wget returns with 8. At the end it seems to be an issue of the proxy server I have to use. It takes 8 long minutes to return :-/

[bjoern.gerhart@wnlpos4-buildsys ~]$ time http_proxy=proxy:81 https_proxy=proxy:81 wget --no-check-certificate -O nvdcve-2.0-2002.xml.gz http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz
--2019-01-09 11:21:27--  http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz
Resolving proxy (proxy)... 10.254.0.18
Connecting to proxy (proxy)|10.254.0.18|:81... connected.
Proxy request sent, awaiting response... 502 cannotconnect
2019-01-09 11:29:29 ERROR 502: cannotconnect.

real    8m2.556s
user    0m0.004s
sys     0m0.005s
[bjoern.gerhart@wnlpos4-buildsys ~]$ echo $?
8
socketpair commented 4 years ago

In my case HTTP response is 410 (!) and exactly the same effect

siebrand commented 4 years ago

The root cause for the HTTP 410 response is https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement 👍

Due to the recent lapse in appropriations and requests from multiple external parties, the NVD has re-evaluated the timeline to end support for the XML 2.0 and 1.2.1 vulnerability data feeds. Previously we had planned to permanently discontinue the XML feeds in April of 2019. However, after consideration we have decided to extend support of these data feeds until October 9th of 2019. After that date the XML feeds will no longer be updated or hosted by the NVD and any new information will only be published in the JSON vulnerability data feeds.

https://nvd.nist.gov/vuln/data-feeds still shows downloadable JSON files with the pattern https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-YYYY.json.gz