New output format: accelerated datamodel query in a savedsearches.conf file

[0x616c6578]

This PR adds a new output format: savedsearches_accelerated_data_model. This combines the savedsearches and data_model formats, with the following changes:

• The data_model query will only return results from summarised data.
• The query settings defined by savedsearches have been extended, to include all settings added in the Splunk Save As -> Alert interface.
• The Hashes Sysmon field has also been mapped to the Processes.process_hash field in the Endpoint.Processes dataset.

Tests have been added for this new format and field mapping.

---

The intent behind this new format is to allow the creation of efficient searches for use in alerting. For example:

[Windows Defender Definition Files Removed]
alert.severity = 5
alert.suppress = 0
alert.track = 1
counttype = number of events
cron_schedule = 0 * * * *
description = Adversaries may disable security tools to avoid possible detection of their tools and activities by removing Windows Defender Definition Files
dispatch.earliest_time = -65m@m
dispatch.latest_time = -5m@m
display.events.fields = ["host", "tag::eventtype"]
display.events.type = raw
display.general.type = statistics
display.page.search.mode = fast
display.page.search.tab = statistics
enableSched = 1
quantity = 0
relation = greater than
search = | tstats summariesonly=true fillnull_value="null" count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_path="*\\MpCmdRun.exe" OR Processes.original_file_name="MpCmdRun.exe" Processes.process="* -RemoveDefinitions*" Processes.process="* -All*" by Processes.process Processes.dest Processes.process_current_directory Processes.process_path Processes.process_integrity_level Processes.original_file_name Processes.parent_process Processes.parent_process_path Processes.parent_process_guid Processes.parent_process_id Processes.process_guid Processes.process_id Processes.user | `drop_dm_object_name(Processes)` | convert timeformat="%F %X" ctime(firstTime) ctime(lastTime)

The savedsearch is a representation of this rule: https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_mpcmdrun_remove_windows_defender_definition.yml.

[0x616c6578]

Closing temporarily. This will be re-requested at a later date with additional lines in the stanza to support Splunk ES Notables.

[thomaspatzke]

Hi! Thanks for your PR. There's a new development that likely makes it obsolete, here's a blog post about it I've published recently. Basically, processing pipelines now allow to embed generated queries into templates to generate custom output formats.

The problem with creating an output format for such things is that he generated output often doesn't fits to the requirements of the people who want to use it. To make such a format universally usable, most of the output has to be parameterized. But as feedback for the savedsearches output of the Splunk backend has shown there's also lot of stuff that is missing or people want to do it completely different. Examples are scheduling, search windows, stanza naming and lots more. In the end there are so many variables that it's easier to create a template than building something that needs to be configured heavily to be usable.