search

stevespringett / nist-data-mirror

A simple Java command-line utility to mirror the CVE JSON data from NIST.
Apache License 2.0
206 stars 93 forks source link

Container running sspringett/nvdmirror:1.3.0 seems to fail in recent updates. #22

Closed Muzamri closed 5 years ago

Muzamri commented 5 years ago

Hi,

I am running the mirror via sspringett/nvdmirror:1.3.0. It was running fine at first, but it seems to not function properly now (i.e the cron task is running on schedule, but it constantly exits with a failure).

This is the log file from the container:

Jun  6 07:26:01 36971edcecc1 crond: crond 4.5 dillon's cron daemon, started with loglevel notice
Jun  6 08:00:16 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-1.0-2002.meta
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.json
Jun  6 12:00:16 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  6 16:00:11 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  6 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  7 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  7 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  7 08:00:16 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  7 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  7 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  7 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 12 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 12 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1

Listing of files in the temp directory used by the mirror:

/tmp/nvd # ls -hlrt
total 2758696
-rw-r--r--    1 root     root      600.7K Jun  6 07:20 nvdcve-2004.xml.gz
-rw-r--r--    1 root     root        4.6M Jun  6 07:20 nvdcve-2004.xml
-rw-r--r--    1 root     root      848.2K Jun  6 07:20 nvdcve-2.0-2004.xml.gz
-rw-r--r--    1 root     root       11.3M Jun  6 07:20 nvdcve-2.0-2004.xml
-rw-r--r--    1 root     root      961.4K Jun  6 07:20 nvdcve-1.0-2004.json.gz
-rw-r--r--    1 root     root       15.0M Jun  6 07:20 nvdcve-1.0-2004.json
-rw-r--r--    1 root     root      938.7K Jun  6 07:20 nvdcve-2005.xml.gz
-rw-r--r--    1 root     root        7.3M Jun  6 07:20 nvdcve-2005.xml
-rw-r--r--    1 root     root        1.3M Jun  6 07:20 nvdcve-2.0-2005.xml.gz
-rw-r--r--    1 root     root       17.4M Jun  6 07:20 nvdcve-2.0-2005.xml
-rw-r--r--    1 root     root        1.5M Jun  6 07:20 nvdcve-1.0-2005.json.gz
-rw-r--r--    1 root     root       23.1M Jun  6 07:20 nvdcve-1.0-2005.json
-rw-r--r--    1 root     root        1.5M Jun  6 07:20 nvdcve-2006.xml.gz
-rw-r--r--    1 root     root       12.1M Jun  6 07:20 nvdcve-2006.xml
-rw-r--r--    1 root     root        2.0M Jun  6 07:20 nvdcve-2.0-2006.xml.gz
-rw-r--r--    1 root     root       27.3M Jun  6 07:20 nvdcve-2.0-2006.xml
-rw-r--r--    1 root     root        2.3M Jun  6 07:21 nvdcve-1.0-2006.json.gz
-rw-r--r--    1 root     root       34.7M Jun  6 07:21 nvdcve-1.0-2006.json
-rw-r--r--    1 root     root        1.5M Jun  6 07:21 nvdcve-2007.xml.gz
-rw-r--r--    1 root     root       11.2M Jun  6 07:21 nvdcve-2007.xml
-rw-r--r--    1 root     root        1.9M Jun  6 07:21 nvdcve-2.0-2007.xml.gz
-rw-r--r--    1 root     root       25.0M Jun  6 07:21 nvdcve-2.0-2007.xml
-rw-r--r--    1 root     root        2.3M Jun  6 07:21 nvdcve-1.0-2007.json.gz
-rw-r--r--    1 root     root       33.1M Jun  6 07:21 nvdcve-1.0-2007.json
-rw-r--r--    1 root     root        1.3M Jun  6 07:21 nvdcve-2009.xml.gz
-rw-r--r--    1 root     root       11.3M Jun  6 07:21 nvdcve-2009.xml
-rw-r--r--    1 root     root        2.1M Jun  6 07:21 nvdcve-2.0-2009.xml.gz
-rw-r--r--    1 root     root       30.6M Jun  6 07:21 nvdcve-2.0-2009.xml
-rw-r--r--    1 root     root        2.4M Jun  6 07:21 nvdcve-1.0-2009.json.gz
-rw-r--r--    1 root     root       42.5M Jun  6 07:21 nvdcve-1.0-2009.json
-rw-r--r--    1 root     root        1.3M Jun  6 07:21 nvdcve-2010.xml.gz
-rw-r--r--    1 root     root       14.9M Jun  6 07:21 nvdcve-2010.xml
-rw-r--r--    1 root     root        2.8M Jun  6 07:21 nvdcve-2.0-2010.xml.gz
-rw-r--r--    1 root     root       45.6M Jun  6 07:21 nvdcve-2.0-2010.xml
-rw-r--r--    1 root     root        3.2M Jun  6 07:22 nvdcve-1.0-2010.json.gz
-rw-r--r--    1 root     root       67.0M Jun  6 07:22 nvdcve-1.0-2010.json
-rw-r--r--    1 root     root      159.9M Jun  6 08:00 nvdcve-1.0-2017.json
-rw-r--r--    1 root     root        5.5M Jun  6 08:01 nvdcve-2018.xml.gz
-rw-r--r--    1 root     root       61.4M Jun  6 08:01 nvdcve-2018.xml
-rw-r--r--    1 root     root       11.8M Jun  6 08:01 nvdcve-2.0-2018.xml.gz
-rw-r--r--    1 root     root      215.6M Jun  6 08:01 nvdcve-2.0-2018.xml
-rw-r--r--    1 root     root        8.4M Jun  6 08:01 nvdcve-1.0-2018.json.gz
-rw-r--r--    1 root     root      209.4M Jun  6 08:01 nvdcve-1.0-2018.json
-rw-r--r--    1 root     root        1.7M Jun  6 08:01 nvdcve-2019.xml.gz
-rw-r--r--    1 root     root       20.5M Jun  6 08:01 nvdcve-2019.xml
-rw-r--r--    1 root     root        3.7M Jun  6 08:01 nvdcve-2.0-2019.xml.gz
-rw-r--r--    1 root     root       71.7M Jun  6 08:01 nvdcve-2.0-2019.xml
-rw-r--r--    1 root     root        2.6M Jun  6 08:01 nvdcve-1.0-2019.json.gz
-rw-r--r--    1 root     root       63.9M Jun  6 08:02 nvdcve-1.0-2019.json
-rw-r--r--    1 root     root        1.1M Jun  6 12:00 nvdcve-2002.xml.gz
-rw-r--r--    1 root     root        8.4M Jun  6 12:00 nvdcve-2002.xml
-rw-r--r--    1 root     root        1.4M Jun  6 12:00 nvdcve-2.0-2002.xml.gz
-rw-r--r--    1 root     root       18.7M Jun  6 12:00 nvdcve-2.0-2002.xml
-rw-r--r--    1 root     root        1.6M Jun  6 12:00 nvdcve-1.0-2002.json.gz
-rw-r--r--    1 root     root       24.7M Jun  6 12:00 nvdcve-1.0-2002.json
-rw-r--r--    1 root     root      312.8K Jun  6 12:00 nvdcve-2003.xml.gz
-rw-r--r--    1 root     root        2.3M Jun  6 12:00 nvdcve-2003.xml
-rw-r--r--    1 root     root      428.2K Jun  6 12:00 nvdcve-2.0-2003.xml.gz
-rw-r--r--    1 root     root        5.4M Jun  6 12:00 nvdcve-2.0-2003.xml
-rw-r--r--    1 root     root      476.2K Jun  6 12:00 nvdcve-1.0-2003.json.gz
-rw-r--r--    1 root     root        6.9M Jun  6 12:00 nvdcve-1.0-2003.json
-rw-r--r--    1 root     root        1.5M Jun  6 12:00 nvdcve-2008.xml.gz
-rw-r--r--    1 root     root       12.8M Jun  6 12:00 nvdcve-2008.xml
-rw-r--r--    1 root     root        2.2M Jun  6 12:00 nvdcve-2.0-2008.xml.gz
-rw-r--r--    1 root     root       30.9M Jun  6 12:00 nvdcve-2.0-2008.xml
-rw-r--r--    1 root     root        2.5M Jun  6 12:00 nvdcve-1.0-2008.json.gz
-rw-r--r--    1 root     root       40.6M Jun  6 12:00 nvdcve-1.0-2008.json
-rw-r--r--    1 root     root        3.0M Jun  6 12:00 nvdcve-2011.xml.gz
-rw-r--r--    1 root     root       29.8M Jun  6 12:00 nvdcve-2011.xml
-rw-r--r--    1 root     root        6.1M Jun  6 12:00 nvdcve-2.0-2011.xml.gz
-rw-r--r--    1 root     root      108.8M Jun  6 12:00 nvdcve-2.0-2011.xml
-rw-r--r--    1 root     root        6.6M Jun  6 12:00 nvdcve-1.0-2011.json.gz
-rw-r--r--    1 root     root      182.1M Jun  6 12:00 nvdcve-1.0-2011.json
-rw-r--r--    1 root     root        1.2M Jun  6 12:00 nvdcve-2012.xml.gz
-rw-r--r--    1 root     root       14.3M Jun  6 12:00 nvdcve-2012.xml
-rw-r--r--    1 root     root        2.5M Jun  6 12:00 nvdcve-2.0-2012.xml.gz
-rw-r--r--    1 root     root       43.2M Jun  6 12:00 nvdcve-2.0-2012.xml
-rw-r--r--    1 root     root        2.9M Jun  6 12:00 nvdcve-1.0-2012.json.gz
-rw-r--r--    1 root     root        1.3M Jun  6 12:00 nvdcve-2013.xml.gz
-rw-r--r--    1 root     root       61.9M Jun  6 12:00 nvdcve-1.0-2012.json
-rw-r--r--    1 root     root       15.1M Jun  6 12:00 nvdcve-2013.xml
-rw-r--r--    1 root     root        2.6M Jun  6 12:00 nvdcve-2.0-2013.xml.gz
-rw-r--r--    1 root     root       43.5M Jun  6 12:00 nvdcve-2.0-2013.xml
-rw-r--r--    1 root     root        3.0M Jun  6 12:00 nvdcve-1.0-2013.json.gz
-rw-r--r--    1 root     root       61.6M Jun  6 12:00 nvdcve-1.0-2013.json
-rw-r--r--    1 root     root        1.6M Jun  6 12:00 nvdcve-2014.xml.gz
-rw-r--r--    1 root     root       16.5M Jun  6 12:00 nvdcve-2014.xml
-rw-r--r--    1 root     root        2.7M Jun  6 12:00 nvdcve-2.0-2014.xml.gz
-rw-r--r--    1 root     root       43.3M Jun  6 12:00 nvdcve-2.0-2014.xml
-rw-r--r--    1 root     root        3.0M Jun  6 12:00 nvdcve-1.0-2014.json.gz
-rw-r--r--    1 root     root       59.6M Jun  6 12:00 nvdcve-1.0-2014.json
-rw-r--r--    1 root     root        1.5M Jun  6 12:00 nvdcve-2015.xml.gz
-rw-r--r--    1 root     root       15.8M Jun  6 12:00 nvdcve-2015.xml
-rw-r--r--    1 root     root        2.3M Jun  6 12:00 nvdcve-2.0-2015.xml.gz
-rw-r--r--    1 root     root       39.4M Jun  6 12:00 nvdcve-2.0-2015.xml
-rw-r--r--    1 root     root        2.6M Jun  6 12:00 nvdcve-1.0-2015.json.gz
-rw-r--r--    1 root     root       52.4M Jun  6 12:00 nvdcve-1.0-2015.json
-rw-r--r--    1 root     root         165 Jun  7 04:00 nvdcve-1.0-2019.meta
-rw-r--r--    1 root     root         166 Jun  7 04:00 nvdcve-1.0-2018.meta
-rw-r--r--    1 root     root         162 Jun  7 08:00 nvdcve-1.0-modified.meta
-rw-r--r--    1 root     root      296.6K Jun  7 08:00 nvdcve-modified.xml.gz
-rw-r--r--    1 root     root        2.7M Jun  7 08:00 nvdcve-modified.xml
-rw-r--r--    1 root     root      541.8K Jun  7 08:00 nvdcve-2.0-modified.xml.gz
-rw-r--r--    1 root     root        9.1M Jun  7 08:00 nvdcve-2.0-modified.xml
-rw-r--r--    1 root     root      432.3K Jun  7 08:00 nvdcve-1.0-modified.json.gz
-rw-r--r--    1 root     root        8.9M Jun  7 08:00 nvdcve-1.0-modified.json
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2006.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2005.meta
-rw-r--r--    1 root     root         163 Jun  7 08:00 nvdcve-1.0-2004.meta
-rw-r--r--    1 root     root         162 Jun  7 08:00 nvdcve-1.0-2003.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2002.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2010.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2009.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2008.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2007.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2014.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2013.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2012.meta
-rw-r--r--    1 root     root         166 Jun  7 08:00 nvdcve-1.0-2011.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2016.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2015.meta
-rw-r--r--    1 root     root        1.7M Jun  7 08:00 nvdcve-2016.xml.gz
-rw-r--r--    1 root     root       20.0M Jun  7 08:00 nvdcve-2016.xml
-rw-r--r--    1 root     root        3.3M Jun  7 08:00 nvdcve-2.0-2016.xml.gz
-rw-r--r--    1 root     root       54.6M Jun  7 08:00 nvdcve-2.0-2016.xml
-rw-r--r--    1 root     root        3.4M Jun  7 08:00 nvdcve-1.0-2016.json.gz
-rw-r--r--    1 root     root       71.7M Jun  7 08:00 nvdcve-1.0-2016.json
-rw-r--r--    1 root     root         166 Jun  7 08:00 nvdcve-1.0-2017.meta
-rw-r--r--    1 root     root        4.1M Jun  7 08:00 nvdcve-2017.xml.gz
-rw-r--r--    1 root     root       46.1M Jun  7 08:00 nvdcve-2017.xml
-rw-r--r--    1 root     root        8.8M Jun  7 08:00 nvdcve-2.0-2017.xml.gz
-rw-r--r--    1 root     root      160.1M Jun  7 08:00 nvdcve-2.0-2017.xml
-rw-r--r--    1 root     root        4.5M Jun  7 08:01 nvdcve-1.0-2017.json.gz

Using wget to download works ok (just confirm it's not my network issue):

/dev/shm # wget https://nvd.nist.gov/feeds/xml/cve/1.2/nvdcve-modified.xml.gz
Connecting to nvd.nist.gov (18.235.227.114:443)
nvdcve-modified.xml. 100% |***********************************************************************************************************************************************|  282k  0:00:00 ETA

I understand that NVD does not have unlimited bandwidth and will throttle or block.

Is there any workaround I can use for a more reliable mirroring?

Thank you for your time.

chris-gillatt commented 5 years ago

I use nist-data-mirror to get my artefacts into a jFrog Artifactory instance (Nexus OSS also works) and then point my builds at that where I can control the reliablity and bandwidth. I do that by running a docker container which contains the jFrog CLI and then schedule it through my CI/CD tool to run every night. The cli will skip failures and try again later, but the previous files will always exist if something was to go majorly wrong meaning I always have a level of cover.

Muzamri commented 5 years ago

Hi @chris-gillatt,

Thank you for sharing your setup.

If I understand correctly -- you used the nist-data-mirror to get the files from NVD and then move it to jFrog to actually serve the file?

I actually have no problem using dependency-check.sh to mirror from the container -- the problem is getting the mirror updated from the actual NVD servers.

It's kind of intermittent, for example, I went ahead and restarted the container:

$ docker logs -f 6199f155757e
Starting mirror
/tmp/nvd/nvdcve-1.0-modified.meta
<...trimmed...>
/tmp/nvd/nvdcve-1.0-2019.json
Mirror started
Jun 12 07:53:58 6199f155757e crond: crond 4.5 dillon's cron daemon, started with loglevel notice
AH00557: httpd: apr_sockaddr_info_get() failed for 6199f155757e
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
AH00557: httpd: apr_sockaddr_info_get() failed for 6199f155757e
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
[Wed Jun 12 07:54:14.015875 2019] [mpm_event:notice] [pid 198:tid 140351906831208] AH00489: Apache/2.4.39 (Unix) configured -- resuming normal operations
[Wed Jun 12 07:54:14.015904 2019] [core:notice] [pid 198:tid 140351906831208] AH00094: Command line: 'httpd -D FOREGROUND'
Jun 12 08:00:16 6199f155757e crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-1.0-2002.meta
<...trimmed...>
/tmp/nvd/nvdcve-1.0-2019.meta

and list the files in the temp directory again:

/tmp/nvd # ls -hlrt
total 2776720
-rw-r--r--    1 root     root      228.3K Jun 12 07:47 nvdcve-modified.xml.gz
-rw-r--r--    1 root     root        2.2M Jun 12 07:47 nvdcve-modified.xml
-rw-r--r--    1 root     root      407.6K Jun 12 07:48 nvdcve-2.0-modified.xml.gz
-rw-r--r--    1 root     root        7.3M Jun 12 07:48 nvdcve-2.0-modified.xml
-rw-r--r--    1 root     root      350.9K Jun 12 07:48 nvdcve-1.0-modified.json.gz
-rw-r--r--    1 root     root        7.5M Jun 12 07:48 nvdcve-1.0-modified.json
-rw-r--r--    1 root     root        1.1M Jun 12 07:48 nvdcve-2002.xml.gz
-rw-r--r--    1 root     root        8.4M Jun 12 07:48 nvdcve-2002.xml
-rw-r--r--    1 root     root        1.4M Jun 12 07:48 nvdcve-2.0-2002.xml.gz
-rw-r--r--    1 root     root       18.7M Jun 12 07:48 nvdcve-2.0-2002.xml
-rw-r--r--    1 root     root        1.6M Jun 12 07:48 nvdcve-1.0-2002.json.gz
-rw-r--r--    1 root     root       24.7M Jun 12 07:48 nvdcve-1.0-2002.json
-rw-r--r--    1 root     root      312.8K Jun 12 07:48 nvdcve-2003.xml.gz
-rw-r--r--    1 root     root        2.3M Jun 12 07:48 nvdcve-2003.xml
-rw-r--r--    1 root     root      428.2K Jun 12 07:48 nvdcve-2.0-2003.xml.gz
-rw-r--r--    1 root     root        5.4M Jun 12 07:48 nvdcve-2.0-2003.xml
-rw-r--r--    1 root     root      600.7K Jun 12 07:48 nvdcve-2004.xml.gz
-rw-r--r--    1 root     root        4.6M Jun 12 07:48 nvdcve-2004.xml
-rw-r--r--    1 root     root      476.2K Jun 12 07:48 nvdcve-1.0-2003.json.gz
-rw-r--r--    1 root     root        6.9M Jun 12 07:48 nvdcve-1.0-2003.json
-rw-r--r--    1 root     root      848.2K Jun 12 07:48 nvdcve-2.0-2004.xml.gz
-rw-r--r--    1 root     root       11.3M Jun 12 07:48 nvdcve-2.0-2004.xml
-rw-r--r--    1 root     root      961.4K Jun 12 07:48 nvdcve-1.0-2004.json.gz
-rw-r--r--    1 root     root       15.0M Jun 12 07:48 nvdcve-1.0-2004.json
-rw-r--r--    1 root     root      938.7K Jun 12 07:48 nvdcve-2005.xml.gz
-rw-r--r--    1 root     root        7.3M Jun 12 07:48 nvdcve-2005.xml
-rw-r--r--    1 root     root        1.3M Jun 12 07:48 nvdcve-2.0-2005.xml.gz
-rw-r--r--    1 root     root       17.4M Jun 12 07:48 nvdcve-2.0-2005.xml
-rw-r--r--    1 root     root        1.5M Jun 12 07:48 nvdcve-1.0-2005.json.gz
-rw-r--r--    1 root     root       23.1M Jun 12 07:48 nvdcve-1.0-2005.json
-rw-r--r--    1 root     root        1.5M Jun 12 07:48 nvdcve-2006.xml.gz
-rw-r--r--    1 root     root       12.1M Jun 12 07:48 nvdcve-2006.xml
-rw-r--r--    1 root     root        2.0M Jun 12 07:48 nvdcve-2.0-2006.xml.gz
-rw-r--r--    1 root     root       27.3M Jun 12 07:48 nvdcve-2.0-2006.xml
-rw-r--r--    1 root     root        2.3M Jun 12 07:48 nvdcve-1.0-2006.json.gz
-rw-r--r--    1 root     root       34.7M Jun 12 07:48 nvdcve-1.0-2006.json
-rw-r--r--    1 root     root        1.5M Jun 12 07:48 nvdcve-2007.xml.gz
-rw-r--r--    1 root     root       11.2M Jun 12 07:48 nvdcve-2007.xml
-rw-r--r--    1 root     root        2.0M Jun 12 07:48 nvdcve-2.0-2007.xml.gz
-rw-r--r--    1 root     root       25.2M Jun 12 07:48 nvdcve-2.0-2007.xml
-rw-r--r--    1 root     root        2.3M Jun 12 07:48 nvdcve-1.0-2007.json.gz
-rw-r--r--    1 root     root       33.1M Jun 12 07:48 nvdcve-1.0-2007.json
-rw-r--r--    1 root     root        1.5M Jun 12 07:48 nvdcve-2008.xml.gz
-rw-r--r--    1 root     root       12.8M Jun 12 07:48 nvdcve-2008.xml
-rw-r--r--    1 root     root        2.2M Jun 12 07:48 nvdcve-2.0-2008.xml.gz
-rw-r--r--    1 root     root       30.9M Jun 12 07:48 nvdcve-2.0-2008.xml
-rw-r--r--    1 root     root        2.5M Jun 12 07:49 nvdcve-1.0-2008.json.gz
-rw-r--r--    1 root     root       40.6M Jun 12 07:49 nvdcve-1.0-2008.json
-rw-r--r--    1 root     root        1.3M Jun 12 07:49 nvdcve-2009.xml.gz
-rw-r--r--    1 root     root       11.3M Jun 12 07:49 nvdcve-2009.xml
-rw-r--r--    1 root     root        2.1M Jun 12 07:49 nvdcve-2.0-2009.xml.gz
-rw-r--r--    1 root     root       30.6M Jun 12 07:49 nvdcve-2.0-2009.xml
-rw-r--r--    1 root     root        2.4M Jun 12 07:49 nvdcve-1.0-2009.json.gz
-rw-r--r--    1 root     root       42.5M Jun 12 07:49 nvdcve-1.0-2009.json
-rw-r--r--    1 root     root        1.3M Jun 12 07:49 nvdcve-2010.xml.gz
-rw-r--r--    1 root     root       14.9M Jun 12 07:49 nvdcve-2010.xml
-rw-r--r--    1 root     root        2.8M Jun 12 07:49 nvdcve-2.0-2010.xml.gz
-rw-r--r--    1 root     root       45.6M Jun 12 07:49 nvdcve-2.0-2010.xml
-rw-r--r--    1 root     root        3.2M Jun 12 07:49 nvdcve-1.0-2010.json.gz
-rw-r--r--    1 root     root       67.0M Jun 12 07:49 nvdcve-1.0-2010.json
-rw-r--r--    1 root     root        3.0M Jun 12 07:49 nvdcve-2011.xml.gz
-rw-r--r--    1 root     root       29.8M Jun 12 07:49 nvdcve-2011.xml
-rw-r--r--    1 root     root        6.1M Jun 12 07:50 nvdcve-2.0-2011.xml.gz
-rw-r--r--    1 root     root      108.8M Jun 12 07:50 nvdcve-2.0-2011.xml
-rw-r--r--    1 root     root        6.6M Jun 12 07:50 nvdcve-1.0-2011.json.gz
-rw-r--r--    1 root     root      182.1M Jun 12 07:50 nvdcve-1.0-2011.json
-rw-r--r--    1 root     root        1.2M Jun 12 07:50 nvdcve-2012.xml.gz
-rw-r--r--    1 root     root       14.3M Jun 12 07:50 nvdcve-2012.xml
-rw-r--r--    1 root     root        2.5M Jun 12 07:50 nvdcve-2.0-2012.xml.gz
-rw-r--r--    1 root     root       43.2M Jun 12 07:50 nvdcve-2.0-2012.xml
-rw-r--r--    1 root     root        2.9M Jun 12 07:50 nvdcve-1.0-2012.json.gz
-rw-r--r--    1 root     root       61.9M Jun 12 07:50 nvdcve-1.0-2012.json
-rw-r--r--    1 root     root        1.3M Jun 12 07:50 nvdcve-2013.xml.gz
-rw-r--r--    1 root     root       15.2M Jun 12 07:50 nvdcve-2013.xml
-rw-r--r--    1 root     root        2.7M Jun 12 07:50 nvdcve-2.0-2013.xml.gz
-rw-r--r--    1 root     root       44.1M Jun 12 07:50 nvdcve-2.0-2013.xml
-rw-r--r--    1 root     root        3.0M Jun 12 07:50 nvdcve-1.0-2013.json.gz
-rw-r--r--    1 root     root       62.0M Jun 12 07:50 nvdcve-1.0-2013.json
-rw-r--r--    1 root     root        1.6M Jun 12 07:51 nvdcve-2014.xml.gz
-rw-r--r--    1 root     root       16.6M Jun 12 07:51 nvdcve-2014.xml
-rw-r--r--    1 root     root        2.8M Jun 12 07:51 nvdcve-2.0-2014.xml.gz
-rw-r--r--    1 root     root       43.9M Jun 12 07:51 nvdcve-2.0-2014.xml
-rw-r--r--    1 root     root        3.0M Jun 12 07:51 nvdcve-1.0-2014.json.gz
-rw-r--r--    1 root     root       60.0M Jun 12 07:51 nvdcve-1.0-2014.json
-rw-r--r--    1 root     root        1.5M Jun 12 07:51 nvdcve-2015.xml.gz
-rw-r--r--    1 root     root       15.9M Jun 12 07:51 nvdcve-2015.xml
-rw-r--r--    1 root     root        2.3M Jun 12 07:51 nvdcve-2.0-2015.xml.gz
-rw-r--r--    1 root     root       39.8M Jun 12 07:51 nvdcve-2.0-2015.xml
-rw-r--r--    1 root     root        2.6M Jun 12 07:51 nvdcve-1.0-2015.json.gz
-rw-r--r--    1 root     root       52.6M Jun 12 07:51 nvdcve-1.0-2015.json
-rw-r--r--    1 root     root        1.8M Jun 12 07:53 nvdcve-2019.xml.gz
-rw-r--r--    1 root     root       21.4M Jun 12 07:53 nvdcve-2019.xml
-rw-r--r--    1 root     root        3.9M Jun 12 07:53 nvdcve-2.0-2019.xml.gz
-rw-r--r--    1 root     root       75.0M Jun 12 07:53 nvdcve-2.0-2019.xml
-rw-r--r--    1 root     root        2.6M Jun 12 07:53 nvdcve-1.0-2019.json.gz
-rw-r--r--    1 root     root       66.0M Jun 12 07:53 nvdcve-1.0-2019.json
-rw-r--r--    1 root     root         162 Jun 12 08:00 nvdcve-1.0-modified.meta
-rw-r--r--    1 root     root         162 Jun 12 08:00 nvdcve-1.0-2003.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2002.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2007.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2006.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2005.meta
-rw-r--r--    1 root     root         163 Jun 12 08:00 nvdcve-1.0-2004.meta
-rw-r--r--    1 root     root         166 Jun 12 08:00 nvdcve-1.0-2011.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2010.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2009.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2008.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2016.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2015.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2014.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2013.meta
-rw-r--r--    1 root     root         165 Jun 12 08:00 nvdcve-1.0-2012.meta
-rw-r--r--    1 root     root        1.7M Jun 12 08:00 nvdcve-2016.xml.gz
-rw-r--r--    1 root     root       20.0M Jun 12 08:00 nvdcve-2016.xml
-rw-r--r--    1 root     root        3.3M Jun 12 08:00 nvdcve-2.0-2016.xml.gz
-rw-r--r--    1 root     root       54.6M Jun 12 08:00 nvdcve-2.0-2016.xml
-rw-r--r--    1 root     root        3.4M Jun 12 08:00 nvdcve-1.0-2016.json.gz
-rw-r--r--    1 root     root       71.7M Jun 12 08:00 nvdcve-1.0-2016.json
-rw-r--r--    1 root     root         166 Jun 12 08:00 nvdcve-1.0-2017.meta
-rw-r--r--    1 root     root        4.1M Jun 12 08:00 nvdcve-2017.xml.gz
-rw-r--r--    1 root     root       46.2M Jun 12 08:00 nvdcve-2017.xml
-rw-r--r--    1 root     root        8.9M Jun 12 08:00 nvdcve-2.0-2017.xml.gz
-rw-r--r--    1 root     root      160.3M Jun 12 08:00 nvdcve-2.0-2017.xml
-rw-r--r--    1 root     root        6.7M Jun 12 08:00 nvdcve-1.0-2017.json.gz
-rw-r--r--    1 root     root      160.8M Jun 12 08:00 nvdcve-1.0-2017.json
-rw-r--r--    1 root     root         166 Jun 12 08:00 nvdcve-1.0-2018.meta
-rw-r--r--    1 root     root        5.6M Jun 12 08:00 nvdcve-2018.xml.gz
-rw-r--r--    1 root     root       62.4M Jun 12 08:01 nvdcve-2018.xml
-rw-r--r--    1 root     root       12.0M Jun 12 08:01 nvdcve-2.0-2018.xml.gz
-rw-r--r--    1 root     root      220.1M Jun 12 08:01 nvdcve-2.0-2018.xml
-rw-r--r--    1 root     root        8.5M Jun 12 08:01 nvdcve-1.0-2018.json.gz
-rw-r--r--    1 root     root      211.8M Jun 12 08:01 nvdcve-1.0-2018.json
-rw-r--r--    1 root     root         165 Jun 12 08:01 nvdcve-1.0-2019.meta

It seems like it's working as intended.

I'm planning to let the container run for another few more days and see what turns up in the logs.

Thank you again.

chris-gillatt commented 5 years ago

@Muzamri yes your understanding is correct. The source is really unreliable (The NVD servers) which is why I started using the nist-data-mirror in the first place. Getting the files into my company firewall was the easy part - making them a reliable source for owasp at the build stage was the problem for me. nist-data-mirror solves it!

Muzamri commented 5 years ago

Ah, alright -- I have a feeling back then, that the NVD servers are not (that) reliable -- the sequential errors that was logged, got me worried though.

I will be closing this issue.

Thank you again for your time. Cheers!