Feature Request: Only download if resource needs updating

stevespringett / nist-data-mirror

A simple Java command-line utility to mirror the CVE JSON data from NIST.

Apache License 2.0

206 stars 93 forks source link

Feature Request: Only download if resource needs updating #3

Closed dwvisser closed 8 years ago

dwvisser commented 9 years ago

Somehow Dependency Check does this (I haven't examined the source closely enough to see exactly how). It would be a nice optimization, so my cron job doesn't have be wasteful of network resources.

athiasjerome commented 8 years ago

There is room for few optimizations: Use of gzip encoding header http://technet.rapaport.com/Info/RapLink/SampleCode/Java_Webrequest_Example.aspx Check local vs remote filesizes (I've been using a single HEAD request for that but you should use the META files) Ref. https://nvd.nist.gov/download.aspx

stevespringett commented 8 years ago

@athiasjerome gzip compression will not matter since this utility already downloads the gzipped version from nist. Cannot use sha256 hash from meta files because the meta only has a hash of the raw xml. This utility only provides a mirror of the gzipped version and does not uncompress. Using HEAD for the time being. Open a new ticket if additional requests.