General cleanup for Windows Server versions/updates/editions

ghost commented 2 years ago

Description

See commit messages. TL;DR:

Improve cpe_update.py runtime. It now runs in <20 seconds on my machine, compared to 10+ minutes previously
Clean up some windows related CPEs, moving R2 to be a version rather than part of product (and other misc changes)
Expand CPE generation/validation to use all CPE fields. 3.5. For deprecated CPEs, collect and emit all (cpe, date) tuples
Split a fingerprint in two to make rake tests happy
- There's an actual fix that should be done here, but this seems effectively the same for less effort than a "real" fix?

Motivation and Context

The apparent mismatch between the separate extracted os.product tag and the product tag of the CPE in a fingerprint prompted the PR:

    <param pos="0" name="os.product" value="Windows Server 2008 R2"/>
    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>

How Has This Been Tested?

Took these steps, based on CONTRIBUTING.md:

$ rake tests
$ ./bin/recog_verify xml/*.xml | grep "[1-9][0-9]* failures"
$ ./bin/recog_standardize xml/*.xml

All tests passing
recog_standardize didn't add any new identifiers.

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] I have updated the documentation accordingly (or changes are not required).
[x] I have added tests to cover my changes (or new tests are not required).
[x] All new and existing tests passed.

mkienow-r7 commented 2 years ago

@elliot-censys Are there others that appear in this grep that should be updated as well? Do you happen to know the difference between "Windows Server 2008 R2", "Windows Server 2008 Datacenter Edition" and "Windows Server 2008 R2, Datacenter Edition"?

grep "<param pos=\"0\" name=\"os.product\" value=\"Windows Server 2008" xml/*.xml

ghost commented 2 years ago

Sorry, I don't know the difference between those.

Yes, the FPs from that grep should be updated (my grepping skills are poor, because I tried to find these and came up short!)

I'll update the other relevant ones to match this change.

ghost commented 2 years ago

Ah, looks like there are also some Windows Server 2012 R2 here too that should change

mkienow-r7 commented 2 years ago

I believe I figured it out. "Datacenter" should not be grouped in os.product since it is the edition. It should be extracted to os.edition. If you have the time would you please take a look at cleaning those up as well since you pulled the thread on Windows Server 2008? :)

EDIT: R2 DataCenter example from NVD cpe:2.3:o:microsoft:windows_server_2008:r2:-:*:*:datacenter:*:x64:*

ghost commented 2 years ago

Sure, I'd be happy to clean this stuff up. It might be a bit before I get to it; do you mind if I just keep the PR open?

mkienow-r7 commented 2 years ago

No issue keeping the PR open until you get around to the cleanup. Thank you for bringing attention to the os.version improvement and taking on the additional cleanup!

ghost commented 2 years ago

Spent some time pouring through the results of your grep and also this one:

grep "<param pos=\"0\" name=\"os.product\" value=\"Windows Server [0-9]\+ R2" xml/*.xml

and ended up catching issues like this for other years/releases/versions/whatever of Window Server. I think I've gotten to all of them now, and I updated the MR description / commit message to better reflect what I've done.

I also quickly went through matches for Windows Server and found some problems that I fixed. There were obviously a lot of matches to this pattern, so I could have missed something, but I think I left this in a good place.

mkienow-r7 commented 2 years ago

Sample of diff after I ran update_cpes.py:

...
   <fingerprint pattern="^(9.[^-]+(?:-[SP]\d)?)-RedHat-[\w.-]+amzn1$">
@@ -661,7 +660,7 @@
     <param pos="0" name="os.build" value="6.3.9600"/>
     <param pos="0" name="os.product" value="Windows Server 2012"/>
     <param pos="0" name="os.version" value="R2"/>
-    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2012:r2"/>
+    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2012:R2"/>
   </fingerprint>

   <fingerprint pattern="^Microsoft DNS 6.2.9200(?: \(([^)]+)\))?$">
@@ -694,7 +693,7 @@
     <param pos="0" name="os.update" value="Service Pack 1"/>
     <param pos="0" name="os.version" value="R2"/>
     <param pos="0" name="os.build" value="6.1.7601"/>
-    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:r2:sp1"/>
+    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:R2"/>
   </fingerprint>
...

   <!-- This value is a spoofed value. There isn't a publicly available version
@@ -742,7 +741,7 @@
     <param pos="0" name="os.product" value="Windows Server 2008"/>
     <param pos="0" name="os.edition" value="Service Pack 2"/>
     <param pos="0" name="os.build" value="6.0.6003"/>
-    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-:sp2"/>
+    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>
   </fingerprint>
...

--- a/xml/smb_native_os.xml
+++ b/xml/smb_native_os.xml
@@ -58,7 +58,7 @@
     <param pos="0" name="os.product" value="Windows XP"/>
     <param pos="1" name="os.build"/>
     <param pos="2" name="os.update"/>
-    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_xp:"/>
+    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_xp:-"/>
   </fingerprint>
...

   <fingerprint pattern="^Windows \(R\) Web Server 2008 (\d+) (Service Pack \d+)$">
@@ -178,7 +178,7 @@
     <param pos="0" name="os.edition" value="Web"/>
     <param pos="1" name="os.build"/>
     <param pos="2" name="os.update"/>
-    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-:{os.update}:web"/>
+    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>
   </fingerprint>

ghost commented 2 years ago

The os.cpe23 parameter values appear to be added by hand rather than from following the Updating CPEs steps. I ran the script against this branch and it seems update_cpes.py doesn't handle more than vendor, product, version. In the case of version it doesn't convert to lower case. We'll need to look at update_cpes.py enhancements to better support version, and also include update and edition.

I took a stab at enhancing update_cpes.py to support this behavior and added some mappings for the service packs this patch originally added.

Some bits are a little hacky, but it seems to work pretty well.

Sorry for the huge delay on responding to your feedback.

mkienow-r7 commented 2 years ago

@elliot-censys I'm working my way through these changes. The performance improvement is great! I noticed there are a number of fingerprint CPE params removed. For example, a:rapid7:metasploit is a value CPE value, but it was removed. Would you elaborate as to why the CPE params were removed by the script?

rapid7 / recog