CPE and CVE records are very large (200k and 1mil, respectively).
They are also added to/updated daily.
as such running any mode that connects CPE or CVE data results in a massive check of all records for updates unnecessarily, some of which have not been modified in over 20 years (specifically modes cve-cwe, cve-cpe, cpe-groups)
As such it makes sense to add some sort of cursor/time point that the script works from each time.
We should also give user ability to add an earliest_modified_time
--earliest_modified_time YYYY-MM-HHThh:mm:ss
This should mean only the _is_latest=true source objects with a modified time >= to earliest_modified_time in the source collection should be considered for updates. In the CVE example, this would mean only newly updated CVEs since the last run would be considered when creating the relationships (thus meaning 100's of records do not need to be parsed)
There should be no default (put another way, default is current behaviour)
NOTE: this will not work with cpe-groups mode, as software objects do not have a modified time. Also, the creation of the groupings requires an awareness of all software objects (also no point b/c of #12)
CPE and CVE records are very large (200k and 1mil, respectively).
They are also added to/updated daily.
as such running any mode that connects CPE or CVE data results in a massive check of all records for updates unnecessarily, some of which have not been modified in over 20 years (specifically modes cve-cwe, cve-cpe, cpe-groups)
As such it makes sense to add some sort of cursor/time point that the script works from each time.
We should also give user ability to add an earliest_modified_time
This should mean only the
_is_latest=truesource objects with a modified time >= toearliest_modified_timein the source collection should be considered for updates. In the CVE example, this would mean only newly updated CVEs since the last run would be considered when creating the relationships (thus meaning 100's of records do not need to be parsed)There should be no default (put another way, default is current behaviour)
NOTE: this will not work with cpe-groups mode, as software objects do not have a
modifiedtime. Also, the creation of the groupings requires an awareness of all software objects (also no point b/c of #12)