We need to validate that our detections not only work in ES, but are also being parsed correctly by the content versioning service
Code change
Added ContentVersioningService
Enables content versioning
Activates CMS parser
Validates detections against CMS events, reports any missing detections or other validation issues. Validations include:
Counts between cms_main and detections based on YAMLs match
Each detection can be matched to a cms_event
Each cms_event can be matched to a detection
No cms_event is repeated
UUIDs match between cms/detection pairs
Versions match between cms/detection pairs
Full search name matches between cms/detection pairs
Correlationsearch label matches between cms/detection pairs
Sourcetype matches between cms/detection pairs
Returns a list of setup functions which are optionally inserted into the main setup of DetectionTestingInfrastructure
Aggregating errors across all threads (setup, testing, view, view shutdown) and reporting en masse
Broke up setup functions list to allow for optional inclusion of the content versioning setup functions
NOTE: Now we raise any exceptions that crop up during setup, instead of just logging them
Added logic to DetectionTestingInfrastructure to query installed ES version, so we can determine when it is appropriate to run content versioning validation
Content versioning validation is only run when 1) integration testing is enabled, 2) ES 8.0.0+ is installed, and 3) when the testing mode is All
Moved logging utility in CorrelationSearch (useful for debugging) to helper.utils.Utils so it could be re-used in ContentVersioningService
Testing
I've tested locally w/ some quick and dirty edits to the savedsearches.conf file and all seems to work well
Want to test in CI, but currently the changes to dashboards in contentctl preclude me from testing this on the security_content repo as it exists today
Caveats
Note that we now raise exceptions from setup, instead of logging and dropping them; I believe this is desirable
I went back and forth as to whether this validation should happen as part of setup, OR if I should create some new type of test case. I opted for the former as this type of testing doesn't fit our current testing paradigm; validation against the cms_main index should happen in aggregate, a test of the whole package, not individual detections. We do not have an easy way to insert testing at that level presently.
Additionally, this validation probably only needs to be run against a single instance and not necessarily all instances in a single run, but I don't see an easy or clean way to do that as well, and there's no harm or impact on execution time as far as I can tell
Context
Code change
ContentVersioningServicecms_mainand detections based on YAMLs matchcms_eventcms_eventcan be matched to a detectioncms_eventis repeatedDetectionTestingInfrastructureraiseany exceptions that crop up during setup, instead of just logging themDetectionTestingInfrastructureto query installed ES version, so we can determine when it is appropriate to run content versioning validationAllCorrelationSearch(useful for debugging) tohelper.utils.Utilsso it could be re-used inContentVersioningServiceTesting
savedsearches.conffile and all seems to work wellCaveats
cms_mainindex should happen in aggregate, a test of the whole package, not individual detections. We do not have an easy way to insert testing at that level presently.TODOs