Validate Diffs - Githubissues

ndowmon commented 3 years ago

See computed diffs between the old integration & new integration. Notes:

This diff does not have any tenable_scan_report, tenable_scan_finding, tenable_scan_malware, or tenable_scan_unwanted_programs entities or associated relationships.
The id properties were changed from number to string in order to conform to some requirements of jobState.addEntity. I could change them back for posterity, and we could open issues to use createIntegrationEntity() for safety.
The mapper creates _key properties using the source entity ID, rather than source entity key. I've visually confirmed that the only difference between the 8 mapped relationships is that some of the old mapped relationships have a tag.AccountName property, where the new mapped relationships do not.
This diff was generated with two fresh integration instances. I'm not sure if the old integration had behavior that allowed certain findings / reports etc to not be deleted (partial datasets). I would appreciate a review of the old code to ensure I haven't missed any partial datasets in the conversion.

 {
   tenable_user_2232707: {
     properties: {
-      id: 2232707
+      id: "2232707"
     }
   }
   tenable_scan_19: {
     properties: {
-      id: 19
+      id: "19"
     }
   }
   tenable_scan_13: {
     properties: {
-      id: 13
+      id: "13"
     }
   }
   tenable_scan_17: {
     properties: {
-      id: 17
+      id: "17"
     }
   }
   tenable_scan_15: {
     properties: {
-      id: 15
+      id: "15"
     }
   }
   tenable_scan_11: {
     properties: {
-      id: 11
+      id: "11"
     }
   }
   tenable_vulnerability_finding_17_14272_1: {
     properties: {
       targets: [
+        "10.132.0.4"
+        "10.17.0.8"
         "159.89.38.68"
-        "10.17.0.8"
-        "10.132.0.4"
+        "36:25:49:fb:f3:86"
+        "a6:a1:a1:62:35:a6"
         "fe80:0:0:0:3425:49ff:fefb:f386"
         "fe80:0:0:0:a4a1:a1ff:fe62:35a6"
-        "36:25:49:fb:f3:86"
-        "a6:a1:a1:62:35:a6"
       ]
     }
   }
   tenable_vulnerability_finding_17_19506_1: {
     properties: {
       targets: [
+        "10.132.0.4"
+        "10.17.0.8"
         "159.89.38.68"
-        "10.17.0.8"
-        "10.132.0.4"
+        "36:25:49:fb:f3:86"
+        "a6:a1:a1:62:35:a6"
         "fe80:0:0:0:3425:49ff:fefb:f386"
         "fe80:0:0:0:a4a1:a1ff:fe62:35a6"
-        "36:25:49:fb:f3:86"
-        "a6:a1:a1:62:35:a6"
       ]
     }
   }
   tenable_user_2109705: {
     properties: {
-      id: 2109705
+      id: "2109705"
     }
   }
   tenable_user_2250642: {
     properties: {
-      id: 2250642
+      id: "2250642"
     }
   }
   tenable_user_2110185: {
     properties: {
-      id: 2110185
+      id: "2110185"
     }
   }
 }

 {
-  4c823f59-40ba-4558-8268-4d6229dd649a:IS:Person:kenan.warren@jupiterone.com: {
-    _type: "mapped_relationship"
-    _class: "IS"
-    _source: "system-mapper"
-    _fromEntityKey: "tenable_user_2232707"
-    _toEntityKey: "Person:kenan.warren@jupiterone.com"
-    properties: {
-    }
-  }
-  tenable_vulnerability_finding_17_19506_1_tenable_vulnerability_19506: {
-    _type: "tenable_vulnerability_finding_is_vulnerability"
-    _class: "IS"
-    _source: "system-mapper"
-    displayName: "IS"
-    _fromEntityKey: "tenable_vulnerability_finding_17_19506_1"
-    _toEntityKey: "tenable_vulnerability_19506"
-    properties: {
-      assetUuid: "604894a3-b8af-4479-85b4-e00343dbd309"
-      pluginId: 19506
-      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
-      tag.AccountName: "Tenable Nick Test"
-      scanId: 17
-    }
-  }
-  tenable_scan_17_identified_tenable_vulnerability_19506: {
-    _type: "tenable_scan_identified_vulnerability"
-    _class: "IDENTIFIED"
-    _source: "system-mapper"
-    displayName: "IDENTIFIED"
-    _fromEntityKey: "tenable_scan_17"
-    _toEntityKey: "tenable_vulnerability_19506"
-    properties: {
-      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
-      count: 1
-      scanId: 17
-      tag.AccountName: "Tenable Nick Test"
-    }
-  }
-  tenable_scan_17_identified_tenable_vulnerability_14272: {
-    _type: "tenable_scan_identified_vulnerability"
-    _class: "IDENTIFIED"
-    _source: "system-mapper"
-    displayName: "IDENTIFIED"
-    _fromEntityKey: "tenable_scan_17"
-    _toEntityKey: "tenable_vulnerability_14272"
-    properties: {
-      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
-      count: 2
-      scanId: 17
-      tag.AccountName: "Tenable Nick Test"
-    }
-  }
-  tenable_vulnerability_finding_17_14272_1_tenable_vulnerability_14272: {
-    _type: "tenable_vulnerability_finding_is_vulnerability"
-    _class: "IS"
-    _source: "system-mapper"
-    displayName: "IS"
-    _fromEntityKey: "tenable_vulnerability_finding_17_14272_1"
-    _toEntityKey: "tenable_vulnerability_14272"
-    properties: {
-      assetUuid: "604894a3-b8af-4479-85b4-e00343dbd309"
-      pluginId: 14272
-      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
-      tag.AccountName: "Tenable Nick Test"
-      scanId: 17
-    }
-  }
-  d552b241-8d88-45f7-8ddb-4417ffdadb0a:IS:Person:nick.dowmon@lifeomic.com: {
-    _type: "mapped_relationship"
-    _class: "IS"
-    _source: "system-mapper"
-    _fromEntityKey: "tenable_user_2250642"
-    _toEntityKey: "Person:nick.dowmon@lifeomic.com"
-    properties: {
-    }
-  }
-  09058885-2e94-42c1-96d4-027cc0481b55:HOSTS:Vendor:Tenable Cloud: {
-    _type: "mapped_relationship"
-    _class: "HOSTS"
-    _source: "system-mapper"
-    _fromEntityKey: "Vendor:Tenable Cloud"
-    _toEntityKey: "tenable_account_49a7e31b-b2b9-41c2-b0bd-3149141bb648"
-    properties: {
-    }
-  }
-  09058885-2e94-42c1-96d4-027cc0481b55:OWNS:mapper:root: {
-    _type: "mapped_relationship"
-    _class: "OWNS"
-    _source: "system-mapper"
-    _fromEntityKey: "mapper:root"
-    _toEntityKey: "tenable_account_49a7e31b-b2b9-41c2-b0bd-3149141bb648"
-    properties: {
-    }
-  }
+  9b93980f-9869-472b-94e9-e4742fb920b9:IDENTIFIED:tenable_vulnerability_19506:tenable_scan_17_identified_tenable_vulnerability_19506: {
+    _type: "tenable_scan_identified_vulnerability"
+    _class: "IDENTIFIED"
+    _source: "system-mapper"
+    displayName: "IDENTIFIED"
+    _fromEntityKey: "tenable_scan_17"
+    _toEntityKey: "tenable_vulnerability_19506"
+    properties: {
+      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
+      count: 1
+      scanId: 17
+    }
+  }
+  1fb2a48d-bf1b-47be-b855-0922c1fe9485:IS:tenable_vulnerability_19506:tenable_vulnerability_finding_17_19506_1_tenable_vulnerability_19506: {
+    _type: "tenable_vulnerability_finding_is_vulnerability"
+    _class: "IS"
+    _source: "system-mapper"
+    displayName: "IS"
+    _fromEntityKey: "tenable_vulnerability_finding_17_19506_1"
+    _toEntityKey: "tenable_vulnerability_19506"
+    properties: {
+      assetUuid: "604894a3-b8af-4479-85b4-e00343dbd309"
+      pluginId: 19506
+      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
+      scanId: 17
+    }
+  }
+  9b93980f-9869-472b-94e9-e4742fb920b9:IDENTIFIED:tenable_vulnerability_14272:tenable_scan_17_identified_tenable_vulnerability_14272: {
+    _type: "tenable_scan_identified_vulnerability"
+    _class: "IDENTIFIED"
+    _source: "system-mapper"
+    displayName: "IDENTIFIED"
+    _fromEntityKey: "tenable_scan_17"
+    _toEntityKey: "tenable_vulnerability_14272"
+    properties: {
+      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
+      count: 2
+      scanId: 17
+    }
+  }
+  05581a65-f3ef-47ca-a66e-ad158221e48f:IS:tenable_vulnerability_14272:tenable_vulnerability_finding_17_14272_1_tenable_vulnerability_14272: {
+    _type: "tenable_vulnerability_finding_is_vulnerability"
+    _class: "IS"
+    _source: "system-mapper"
+    displayName: "IS"
+    _fromEntityKey: "tenable_vulnerability_finding_17_14272_1"
+    _toEntityKey: "tenable_vulnerability_14272"
+    properties: {
+      assetUuid: "604894a3-b8af-4479-85b4-e00343dbd309"
+      pluginId: 14272
+      scanUuid: "a4c0b1b7-8e87-41e0-bd62-a284264b1b93"
+      scanId: 17
+    }
+  }
+  186b8b60-9911-4e53-b3c7-c8ba78f5d80f:IS:Person:nick.dowmon@lifeomic.com: {
+    _type: "mapped_relationship"
+    _class: "IS"
+    _source: "system-mapper"
+    _fromEntityKey: "tenable_user_2250642"
+    _toEntityKey: "Person:nick.dowmon@lifeomic.com"
+    properties: {
+    }
+  }
+  1e44a074-b7d2-4ca6-b111-e0c4f25a1a23:IS:Person:kenan.warren@jupiterone.com: {
+    _type: "mapped_relationship"
+    _class: "IS"
+    _source: "system-mapper"
+    _fromEntityKey: "tenable_user_2232707"
+    _toEntityKey: "Person:kenan.warren@jupiterone.com"
+    properties: {
+    }
+  }
+  5347517e-a4a4-4ab8-9d64-d0acb628eb81:HOSTS:Vendor:Tenable Cloud: {
+    _type: "mapped_relationship"
+    _class: "HOSTS"
+    _source: "system-mapper"
+    _fromEntityKey: "Vendor:Tenable Cloud"
+    _toEntityKey: "tenable_account_49a7e31b-b2b9-41c2-b0bd-3149141bb648"
+    properties: {
+    }
+  }
+  5347517e-a4a4-4ab8-9d64-d0acb628eb81:OWNS:mapper:root: {
+    _type: "mapped_relationship"
+    _class: "OWNS"
+    _source: "system-mapper"
+    _fromEntityKey: "mapper:root"
+    _toEntityKey: "tenable_account_49a7e31b-b2b9-41c2-b0bd-3149141bb648"
+    properties: {
+    }
+  }
 }

ndowmon commented 3 years ago

@fomentia FYI - I'm sure you're already aware of this behavior but just in case.

The mapper creates _key properties using the source entity ID, rather than source entity key. I've visually confirmed that the only difference between the 8 mapped relationships is that some of the old mapped relationships have a tag.AccountName property, where the new mapped relationships do not.

aiwilliams commented 3 years ago

The id properties were changed from number to string in order to conform to some requirements of jobState.addEntity. I could change them back for posterity, and we could open issues to use createIntegrationEntity() for safety.

I think it good that they become strings. I'd be surprised if anyone is running queries that expect a number.

The mapper creates _key properties using the source entity ID, rather than source entity key. I've visually confirmed that the only difference between the 8 mapped relationships is that some of the old mapped relationships have a tag.AccountName property, where the new mapped relationships do not.

I saw similar results in the Okta integration. As long as the relationships between the entities exist, I would expect any queries that care about the relationships will continue to function.

This diff was generated with two fresh integration instances. I'm not sure if the old integration had behavior that allowed certain findings / reports etc to not be deleted (partial datasets). I would appreciate a review of the old code to ensure I haven't missed any partial datasets in the conversion.

Reviewing the old code, I don't see that there are any partial sets. Does this new code do anything to limit the amount of data processed?

ndowmon commented 3 years ago

Reviewing the old code, I don't see that there are any partial sets. Does this new code do anything to limit the amount of data processed?

@aiwilliams no, not yet. I suspect once this goes into production we'll want to make further incremental improvements there.

JupiterOne-Archives / graph-tenable-io

Validate Diffs #93