cve xml feed broken

[tako774]

CVE 2018 feed xml keeps unable to be parsed. (Maybe #18 is the same)

$ go-exploitdb fetch exploitdb ... WARN[08-02|02:24:01] Failed to Unmarshal XML. URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml err="XML syntax error on line 169238: expected /> in element"

I tried 4 times curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml, but all of them were broken at around 10MB (response body ended with "Virus/Spyware Download Blocked" html message).

Then I tried changing protocol http to https curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml, it was all OK (22MB).

Feed urls in the CVE site use https https://cve.mitre.org/data/downloads/index.html .

Currently go-exploitdb uses http when downloading CVE feeds, but it might be good to use https.

[tako774]

http and https diff test Only CVE 2018 feed xml differs.

mkdir http cd http curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2000.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2001.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2002.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2003.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2004.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2005.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2006.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2007.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2008.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2009.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2010.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2011.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2012.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2013.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2014.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2015.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2016.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2017.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2019.xml

cd .. mkdir https cd https curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2000.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2001.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2002.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2003.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2004.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2005.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2006.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2007.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2008.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2009.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2010.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2011.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2012.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2013.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2014.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2015.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2016.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2017.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2019.xml

cd .. diff -rq http https -> Files http/allitems-cvrf-year-2018.xml and https/allitems-cvrf-year-2018.xml differ

[tako774]

Now xml feed is valid. From my server grep warn /var/log/go-exploitdb/go-exploitdb.log, it had been broken from 2019-06-27T04:28:01+0900 to 2019-08-19T04:25:00+0900.

Anyway, https should be selected to get cve feeds.