Compare CWE, CAPEC & VIA4 last-modified using HEAD

[oh2fih]

Closes https://github.com/cve-search/CveXplore/issues/270

• New source_changed() uses the last-modified already cached in the info collection & compares it with the last-modified of the current source using HTTP HEAD method.
  • If those two are an exact match, return False; the update can be skipped.
  • In any other case including all exceptions assume the source has changed, return True and continue as before.
  • The download_site() already updates the cache so we don't have to worry about that here. The comparison of cached last-modified should now newer give "are not modified" for sources using source_changed(), but is still relevant for sources not using it.
• Use this method for CWE, CAPEC & VIA4 sources.
  • There's a valid reason documented as a comment in download_site() we should keep downloading EPSS.

[oh2fih]

Tested with CVE-Search (+patch https://github.com/cve-search/cve-search/pull/1076) by running ./sbin/db_updater.py -v.

Overall, the total update duration dropped from 0:02:41 to 0:00:21 when these three sources did not change.

• ✅ When the sources haven't changed:

  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CVE database update
  CveXplore.core.database_maintenance.sources_process - INFO     - CWE database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Comparing cached last-modified of CWE (2024-02-29 13:58:32) with URL https://cwe.mitre.org/data/xml/cwec_latest.xml.zip
  CveXplore.core.database_maintenance.sources_process - INFO     - CWE's are not modified since the last update
  CveXplore.core.database_maintenance.sources_process - INFO     - Skipped CWE database update
  CveXplore.core.database_maintenance.sources_process - INFO     - CAPEC database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Comparing cached last-modified of CAPEC (2023-01-24 18:32:31) with URL https://capec.mitre.org/data/xml/capec_latest.xml
  CveXplore.core.database_maintenance.sources_process - INFO     - CAPEC's are not modified since the last update
  CveXplore.core.database_maintenance.sources_process - INFO     - Skipped CAPEC database update
  CveXplore.core.database_maintenance.sources_process - INFO     - VIA4 database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Comparing cached last-modified of VIA4 (2021-01-23 15:47:42) with URL https://www.cve-search.org/feeds/via4.json
  CveXplore.core.database_maintenance.sources_process - INFO     - VIA4's are not modified since the last update
  CveXplore.core.database_maintenance.sources_process - INFO     - Skipped VIA4 database update
  CveXplore.core.database_maintenance.sources_process - INFO     - EPSS database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  . . .
  CveXplore.core.database_maintenance.main_updater - INFO     - Update Total duration: 0:00:21.072967

• ✅ After removing info collection altogether, forcing update of every source:

  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CVE database update
  CveXplore.core.database_maintenance.sources_process - INFO     - CWE database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Cached last-modified for CWE's not found; should update
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:00:05.580364
  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CWE database update
  CveXplore.core.database_maintenance.sources_process - INFO     - CAPEC database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Cached last-modified for CAPEC's not found; should update
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:01:01.550748
  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CAPEC database update
  CveXplore.core.database_maintenance.sources_process - INFO     - VIA4 database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Cached last-modified for VIA4's not found; should update
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:00:52.940457
  CveXplore.core.database_maintenance.sources_process - INFO     - Finished VIA4 database update
  CveXplore.core.database_maintenance.sources_process - INFO     - EPSS database update started

• ✅ Manually set the dates in info collection to be anything else:

  cvedb> db.info.updateOne( { db: "cwe" }, { $set: { lastModified: ISODate('2000-01-01T00:00:00.000Z') } } )
  cvedb> db.info.updateOne( { db: "capec" }, { $set: { lastModified: ISODate('2000-01-01T00:00:00.000Z') } } )
  cvedb> db.info.updateOne( { db: "via4" }, { $set: { lastModified: ISODate('2000-01-01T00:00:00.000Z') } } )

  Result:

  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CVE database update
  CveXplore.core.database_maintenance.sources_process - INFO     - CWE database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Comparing cached last-modified of CWE (2000-01-01 00:00:00) with URL https://cwe.mitre.org/data/xml/cwec_latest.xml.zip
  CveXplore.core.database_maintenance.sources_process - INFO     - CWE's last-modified changed (2024-02-29 13:58:32); should update
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:00:10.001301
  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CWE database update
  CveXplore.core.database_maintenance.sources_process - INFO     - CAPEC database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Comparing cached last-modified of CAPEC (2000-01-01 00:00:00) with URL https://capec.mitre.org/data/xml/capec_latest.xml
  CveXplore.core.database_maintenance.sources_process - INFO     - CAPEC's last-modified changed (2023-01-24 18:32:31); should update
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:00:12.658176
  CveXplore.core.database_maintenance.sources_process - INFO     - Finished CAPEC database update
  CveXplore.core.database_maintenance.sources_process - INFO     - VIA4 database update started
  CveXplore.core.database_maintenance.sources_process - INFO     - Comparing cached last-modified of VIA4 (2000-01-01 00:00:00) with URL https://www.cve-search.org/feeds/via4.json
  CveXplore.core.database_maintenance.sources_process - INFO     - VIA4's last-modified changed (2021-01-23 15:47:42); should update
  CveXplore.core.database_maintenance.sources_process - INFO     - Downloading files (max 10 workers)
  CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:00:58.668708
  CveXplore.core.database_maintenance.sources_process - INFO     - Finished VIA4 database update
  CveXplore.core.database_maintenance.sources_process - INFO     - EPSS database update started
  . . .
  CveXplore.core.database_maintenance.main_updater - INFO     - Update Total duration: 0:02:41.421019