Add support to watcher job to modules

[ph]

Currently, modules support creating machine learning jobs when we run the setup subcommand; the jobs are disabled by default. But users can enable them. We should apply the same logic for creating watcher job.

Tasks:

• [ ] Read and import watcher job from the watcher directory, disabled by default.
• [ ] Add an integration test for them

[andrewkroh]

From a user POV I think it would be really useful to accept a format like sigma for defining what to alert on. Information about who to alert (the action) could be potentially be configured on a global basis or on a per alert basis.

Rule Definition Example (source):

title: Mimikatz Detection LSASS Access
status: experimental
description: Detects process access to LSASS which is typical for Mimikatz (0x1000 PROCESS_QUERY_ LIMITED_INFORMATION, 0x0400 PROCESS_QUERY_ INFORMATION, 0x0010 PROCESS_VM_READ)
references:
    - https://onedrive.live.com/view.aspx?resid=D026B4699190F1E6!2843&ithint=file%2cpptx&app=PowerPoint&authkey=!AMvCRTKB_V1J5ow
logsource:
    product: windows
    service: sysmon
detection:
    selection:
        EventID: 10
        TargetImage: 'C:\windows\system32\lsass.exe'
        GrantedAccess: '0x1410'
    condition: selection

Generated Watch:

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "not_eq": 0
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [],
        "body": {
          "query": {
            "query_string": {
              "query": "(TargetProcess:\"C\\:\\\\Windows\\\\System32\\\\lsass.exe\" AND EventID:\"8\" AND NOT _exists_:StartModule)",
              "analyze_wildcard": true
            }
          },
          "size": 0
        }
      }
    }
  },
  "actions": {
    "logging-action": {
      "logging": {
        "text": "Sigma Rule 'Password Dumper Remote Thread in LSASS': Hits:\n{{#ctx.payload.hits.hits}}{{_source}}\n================================================================================\n{{/ctx.payload.hits.hits}}"
      }
    }
  }
}

Example (source):

title: Microsoft Office Product Spawning Windows Shell
status: experimental
description: Detects a Windows command line executable started from Microsoft Word, Excel, Powerpoint, Publisher and Visio.
references:
    - https://www.hybrid-analysis.com/sample/465aabe132ccb949e75b8ab9c5bda36d80cf2fd503d52b8bad54e295f28bbc21?environmentId=100
author: Michael Haag
logsource:
    product: windows
    service: sysmon
detection:
    selection:
        EventID: 1
        ParentImage:
            - '*\WINWORD.EXE'
            - '*\EXCEL.EXE'
            - '*\POWERPNT.exe'
            - '*\MSPUB.exe'
            - '*\VISIO.exe'
        Image:
            - '*\cmd.exe'
            - '*\powershell.exe'
            - '*\wscript.exe'
            - '*\cscript.exe'
            - '*\sh.exe'
            - '*\bash.exe'
            - '*\scrcons.exe'
            - '*\schtasks.exe'  # see https://www.hybrid-analysis.com/sample/b409538c99f99b94a5035d9fa44a506b41be0feb23e89b7e4d272ba791aa6002?environmentId=100
            - '*\regsvr32.exe'  # see https://twitter.com/subTee/status/899283365647458305
            - '*\hh.exe'  # see https://www.hybrid-analysis.com/sample/6abc2b63f1865a847ff7f5a9d49bb944397b36f5503b9718d6f91f93d60f7cd7?environmentId=100
    condition: selection
fields:
    - CommandLine
    - ParentCommandLine
falsepositives:
    - unknown
level: high

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "not_eq": 0
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [],
        "body": {
          "query": {
            "query_string": {
              "query": "(Image:(\"*\\\\cmd.exe\" \"*\\\\powershell.exe\" \"*\\\\wscript.exe\" \"*\\\\cscript.exe\" \"*\\\\sh.exe\" \"*\\\\bash.exe\" \"*\\\\scrcons.exe\" \"*\\\\schtasks.exe\" \"*\\\\regsvr32.exe\" \"*\\\\hh.exe\") AND EventID:\"1\" AND ParentImage:(\"*\\\\WINWORD.EXE\" \"*\\\\EXCEL.EXE\" \"*\\\\POWERPNT.exe\" \"*\\\\MSPUB.exe\" \"*\\\\VISIO.exe\"))",
              "analyze_wildcard": true
            }
          },
          "size": 0
        }
      }
    }
  },
  "actions": {
    "logging-action": {
      "logging": {
        "text": "Sigma Rule 'Microsoft Office Product Spawning Windows Shell': Hits:\n{{#ctx.payload.hits.hits}}Hit on {{_source.@timestamp}}:\n      CommandLine = {{_source.CommandLine}}\nParentCommandLine = {{_source.ParentCommandLine}}================================================================================\n{{/ctx.payload.hits.hits}}"
      }
    }
  }
}

[ruflin]

An additional approach here could also be to use fields.yml to create some very simple watches. In the system/disk metricset it could be that if a volume is >80% full an alert is triggered. This is much less powerful then what @andrewkroh describes above and I think we should have the ability to provide "complex" watches but also the very simple one to get started which we can automatically generated out of the fields.yml.

[andrewkroh]

On a related note to Sigma, I see that Microsoft is using a query language to find interesting things in their data:

• APT Hunting Queries - https://github.com/Microsoft/WindowsDefenderATP-Hunting-Queries
• Query Language - https://docs.loganalytics.io/docs/Language-Reference/

[ph]

We are currently moving modules to EPM with that we can close this issue.

[ph]

@ruflin something to investigate with EPM, really good comments from @andrewkroh

[ruflin]

It would be nice to have something as described above to create alerts with the new alerting framework in Kibana instead of going with watcher.