rapid7 / recog

Pattern recognition for hosts, services, and content
Other
662 stars 195 forks source link

update_cpes.py: add deprecation info to output #429

Closed tsellers-r7 closed 2 years ago

tsellers-r7 commented 2 years ago

Description

This PR updates the output of the update_cpes.py script so as to indicate when it failed to generate a CPE due to that CPE being deprecated in the NIST official database.

ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product mds_9000 from vendor cisco invalid for CPE o and no mapping.  This combination is DEPRECATED by o:cisco:mds_9000_san-os at 2010-12-28T17:36:45.943Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z

It also fixes SOME but NOT all of the deprecated CPEs.

I recommend looking to see if we can modify the log output to emit the file name in a manner similar to what is done in our other tests. That would make identifying issues across all files MUCH easier.

Motivation and Context

While troubleshooting some CPE changes it became obvious that the effort was much more time consuming that it should have been and probably too much to ask of an outside contributor.

How Has This Been Tested?

Tested by running against the existing codebase using an up to date NIST database. This was done using the instructions in CONTRIBUTING.MD

curl -o official-cpe-dictionary_v2.3.xml.gz https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz && \
gunzip official-cpe-dictionary_v2.3.xml.gz
ls xml/*.xml | parallel --gnu "python update_cpes.py {} official-cpe-dictionary_v2.3.xml cpe-remap.yaml"  2>>errors.txt 
bin/recog_cleanup
$ cat errors.txt | grep DEPRE
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product mds_9000 from vendor cisco invalid for CPE o and no mapping.  This combination is DEPRECATED by o:cisco:mds_9000_san-os at 2010-12-28T17:36:45.943Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product mds_9000 from vendor cisco invalid for CPE o and no mapping.  This combination is DEPRECATED by o:cisco:mds_9000_san-os at 2010-12-28T17:36:45.943Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product nginx from vendor nginx invalid for CPE a and no mapping.  This combination is DEPRECATED by a:f5:nginx at 2021-11-01T16:02:37.240Z
ERROR:root:Product vios from vendor ibm invalid for CPE o and no mapping.  This combination is DEPRECATED by a:ibm:vios at 2021-08-31T15:42:22.903Z
ERROR:root:Product vios from vendor ibm invalid for CPE o and no mapping.  This combination is DEPRECATED by a:ibm:vios at 2021-08-31T15:42:22.903Z

Types of changes

New feature

Checklist: