Sigma analyzer two rules match on same event

[lprat]

Hi,

I tested on sample (https://github.com/google/timesketch/blob/master/test_tools/test_events/sigma_events.jsonl) to match 2 times on same event, with rules sigma (https://github.com/google/timesketch/blob/master/data/sigma/rules/lnx_susp_zenmap.yml) with differents names lnx_susp_zenmap.yml & lnx2_susp_zenmap.yml: Result on log:

[2020-11-03 17:26:24,110] timesketch.tasks/INFO [sigma] result: Applied 2 tags

But in timesketch:

tag:
[
  "sigma_lnx_susp_zenmap"
]

when we should have:

tag:
[
  "sigma_lnx_susp_zenmap",
  "sigma_lnx2_susp_zenmap"
]

An other problem, i would use 'es-dsl' for use regex, but by default is es-qs (however I have deactivated this mode in sigma_config.yaml).

title: Timesketch Sigma config
order: 20
backends:
  - es-dsl
logsources:
...

Log show query es-qs:

[2020-11-03 17:46:19,732] timesketch.analyzers.sigma_tagger/INFO [sigma] Generated query (data_type:("shell\:zsh\:history" OR "bash\:history\:command" OR "apt\:history\:line" OR "selinux\:line") AND "*apt\-get\ install\ zmap*")

when we should have:

{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must": [
            {
              "bool": {
                "should": [
                  {
                    "match_phrase": {
                      "data_type": "shell:zsh:history"
                    }
                  },
                  {
                    "match_phrase": {
                      "data_type": "bash:history:command"
                    }
                  },
                  {
                    "match_phrase": {
                      "data_type": "apt:history:line"
                    }
                  },
                  {
                    "match_phrase": {
                      "data_type": "selinux:line"
                    }
                  }
                ]
              }
            },
            {
              "multi_match": {
                "query": "*apt-get install zmap*",
                "fields": [],
                "type": "phrase"
              }
            }
          ]
        }
      }
    }
  }
}

The problem is that you are fix query type: https://github.com/google/timesketch/blob/master/timesketch/lib/analyzers/sigma_tagger.py#L74 without let's choice (ElasticsearchDSLBackend, ...). I tried to change "ElasticsearchQuerystringBackend" by "ElasticsearchDSLBackend", but nothing matches... I think the query not send correctly for DSL type.

Last question, when you launch query to ES (for sigma), are you limited in number of result returned?

Thank, Lionel

[jaegeral]

Hey, for the first one, I will have to confirm that. Can you show the output of celery when you run it like you did here:

[2020-11-03 17:46:19,732] timesketch.analyzers.sigma_tagger/INFO [sigma] Generated query (data_type:("shell\:zsh\:history" OR "bash\:history\:command" OR "apt\:history\:line" OR "selinux\:line") AND "*apt\-get\ install\ zmap*") ? Have you changed the rule uuid for the second one?

For your second question, can you elaborate on why you would like to use es-dsl? You are correct we are currently only parsing to es-qswe could remove that from the default config. I has been there for a while.

Last question: No there should not be a limit. As you can see in https://github.com/google/timesketch/blob/master/timesketch/lib/analyzers/sigma_tagger.py#L60

I would suggest to make one Github issue per question / Issue to keep it cleaner and also feel free to join our Slack space for further discussions if that is more helpful for you.

[lprat]

Hi,

I tried to change uuid, and same result:

-------------------lnx3_susp_zenmap.yml-----------------
title: Suspicious Installation of Zenmap
id: 5266a592-b793-11ea-b3de-0242ac130005
description: Detects suspicious installation of Zenmap
references:
    - https://rmusser.net/docs/ATT&CK-Stuff/ATT&CK/Discovery.html
author: Alexander Jaeger
date: 2020/06/26
modified: 2020/06/26
logsource:
    product: linux
    service: shell
detection:
    keywords:
        # Generic suspicious commands
        - message: "apt-get install zmap"
    condition: keywords
falsepositives:
    - Unknown
level: high
-------------------------------------------------------------------
-------------------lnx_susp_zenmap.yml-----------------
title: Suspicious Installation of Zenmap
id: 5266a592-b793-11ea-b3de-0242ac130004
description: Detects suspicious installation of Zenmap
references:
    - https://rmusser.net/docs/ATT&CK-Stuff/ATT&CK/Discovery.html
author: Alexander Jaeger
date: 2020/06/26
modified: 2020/06/26
logsource:
    service: shell
detection:
    keywords:
        # Generic suspicious commands
        - '*apt-get install zmap*'
    condition: keywords
falsepositives:
    - Unknown
level: high
-------------------------------------------------------------------
------------------LOG CELERY------------
[2020-11-04 08:37:39,629] celery.pool/DEBUG TaskPool: Apply <function _fast_trace_task at 0x7f2dc8972f28> (args:('timesketch.lib.tasks.run_sketch_analyzer', '52b73a3a-158f-4cc9-8ec4-ee880593cb2d', {'lang': 'py', 
'task': 'timesketch.lib.tasks.run_sketch_analyzer', 'id': '52b73a3a-158f-4cc9-8ec4-ee880593cb2d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '
6009bdbb-f7f6-49d4-b1b0-64ec736903ce', 'parent_id': '6009bdbb-f7f6-49d4-b1b0-64ec736903ce', 'argsrepr': "('dd627756168a47869e7d5022216b1906', 1, 19, 'sigma')", 'kwargsrepr': '{}', 'origin': 'gen155@26fa720cf417',
 'reply_to': '3f4da99c-029f-340f-aa69-c5f4ecea63f2', 'correlation_id': '52b73a3a-158f-4cc9-8ec4-ee880593cb2d', 'hostname': 'celery@26fa720cf417', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priori
ty': 0, 'redelivered': None}, 'args': ['dd627756168a47869e7d5022216b1906', 1, 19, 'sigma'], 'kwargs': {}}, b'[["dd627756168a47869e7d5022216b1906", 1, 19, "sigma"], {}, {"callbacks": null, "errbacks": null, "chain
": [], "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2020-11-04 08:37:39,633] celery.worker.request/DEBUG Task accepted: timesketch.lib.tasks.run_sketch_analyzer[52b73a3a-158f-4cc9-8ec4-ee880593cb2d] pid:155
[2020-11-04 08:37:40,041] timesketch.analyzers.sigma_tagger/DEBUG [sigma] Loading config from /etc/timesketch/sigma_config.yaml
[2020-11-04 08:37:40,099] timesketch.analyzers.sigma_tagger/INFO [sigma] Reading rules from /usr/local/src/timesketch/data/sigma/rules/lnx3_susp_zenmap.yml
[2020-11-04 08:37:40,105] timesketch.analyzers.sigma_tagger/INFO [sigma] Generated query (data_type:("shell\:zsh\:history" OR "bash\:history\:command" OR "apt\:history\:line" OR "selinux\:line") AND message:"apt\
-get\ install\ zmap")
[2020-11-04 08:37:40,106] urllib3.connectionpool/DEBUG Starting new HTTP connection (1): tselastic:9200
[2020-11-04 08:37:40,122] urllib3.connectionpool/DEBUG http://tselastic:9200 "POST /dd627756168a47869e7d5022216b1906/_refresh HTTP/1.1" 200 49
[2020-11-04 08:37:40,130] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 08:37:40,159] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /dd627756168a47869e7d5022216b1906/_search?_source_includes=human_readable%2Ctag%2Cmessage%2C__ts_emojis&scroll=1m&search_type=quer
y_then_fetch HTTP/1.1" 200 545
[2020-11-04 08:37:40,165] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 08:37:40,184] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /_search/scroll?scroll=5m HTTP/1.1" 200 297
[2020-11-04 08:37:40,186] timesketch.analyzers.sigma_tagger/INFO [sigma] Reading rules from /usr/local/src/timesketch/data/sigma/rules/linux/lnx_susp_zenmap.yml
[2020-11-04 08:37:40,191] timesketch.analyzers.sigma_tagger/INFO [sigma] Generated query (data_type:("shell\:zsh\:history" OR "bash\:history\:command" OR "apt\:history\:line" OR "selinux\:line") AND "*apt\-get\ i
nstall\ zmap*")
[2020-11-04 08:37:40,194] urllib3.connectionpool/DEBUG http://tselastic:9200 "POST /dd627756168a47869e7d5022216b1906/_refresh HTTP/1.1" 200 49
[2020-11-04 08:37:40,199] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 08:37:40,223] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /dd627756168a47869e7d5022216b1906/_search?_source_includes=human_readable%2Ctag%2Cmessage%2C__ts_emojis&scroll=1m&search_type=quer
y_then_fetch HTTP/1.1" 200 545
[2020-11-04 08:37:40,226] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 08:37:40,234] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /_search/scroll?scroll=5m HTTP/1.1" 200 296
[2020-11-04 08:37:40,667] urllib3.connectionpool/DEBUG http://tselastic:9200 "POST /_bulk HTTP/1.1" 200 488
[2020-11-04 08:37:40,669] timesketch.tasks/INFO [sigma] result: Applied 2 tags
* lnx3_susp_zenmap: 1
* lnx_susp_zenmap: 1

[2020-11-04 08:37:40,671] celery.app.trace/INFO Task timesketch.lib.tasks.run_sketch_analyzer[52b73a3a-158f-4cc9-8ec4-ee880593cb2d] succeeded in 1.0391080812551081s: 'dd627756168a47869e7d5022216b1906'
-------------------------------------------

-------------CURL VERIFY---------------
curl -X GET "tselastic:9200/dd627756168a47869e7d5022216b1906/_search?pretty" -H 'Content-Type: application/json' -d'
> {
>   "query": {
>     "query_string": {
>       "query": "message:\"apt-get\"",
>       "default_field": "message"
>     }
>   }
> }
> '
{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.117401,
    "hits" : [
      {
        "_index" : "dd627756168a47869e7d5022216b1906",
        "_type" : "_doc",
        "_id" : "iGuujnUBnpFBqXiee6Kr",
        "_score" : 1.117401,
        "_source" : {
          "message" : "Install: zmap:amd64 (1.1.0-1) [Commandline: apt-get install zmap]",
          "timestamp" : 123456791,
          "datetime" : "2015-07-24T19:01:03+00:00",
          "timestamp_desc" : "foo",
          "command" : "Commandline: apt-get install zmap",
          "data_type" : "apt:history:line",
          "display_name" : "GZIP:/var/log/apt/history.log.1.gz",
          "filename" : "/var/log/apt/history.log.1.gz",
          "packages" : "Install: zmap:amd64 (1.1.0-1)",
          "parser" : "apt_history",
          "tag" : [
            "sigma_lnx_susp_zenmap"
          ]
        }
      }
    ]
  }
}

---------------------------------------

For question on ES-QS vs ES-DSL, I thought it was not possible to create a query regex with es-qs (but I was wrong: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_regular_expressions). Howerver, my query regexp does not work (query: "message:/insta[a-z]{2} zmap/"). When I use the query: "message:/insta[a-z]{2}/", it works, but if you put more than one word then it doesn't work... First problem is "https://github.com/google/timesketch/blob/cdf0275dc0314e14385797c68a8c774f50be3344/timesketch/lib/analyzers/sigma_tagger.py#L124", you remove .keyword (raw format, no splitted string).

curl -X GET "tselastic:9200/dd627756168a47869e7d5022216b1906/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
      "constant_score": {
        "filter": {
          "bool": {
            "must": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "data_type": "shell:zsh:history"
                      }
                    },
                    {
                      "match_phrase": {
                        "data_type": "bash:history:command"
                      }
                    },
                    {
                      "match_phrase": {
                        "data_type": "apt:history:line"
                      }
                    },
                    {
                      "match_phrase": {
                        "data_type": "selinux:line"
                      }
                    }
                  ]
                }
              },
              {
                "regexp": {
                  "message.keyword": {
                      "value": ".*apt\\-get install zmap.*"
                  }
                }
              }
            ]
          }
        }
      }
    }
}
'
{
  "took" : 13,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "dd627756168a47869e7d5022216b1906",
        "_type" : "_doc",
        "_id" : "iGuujnUBnpFBqXiee6Kr",
        "_score" : 1.0,
        "_source" : {
          "message" : "Install: zmap:amd64 (1.1.0-1) [Commandline: apt-get install zmap]",
          "timestamp" : 123456791,
          "datetime" : "2015-07-24T19:01:03+00:00",
          "timestamp_desc" : "foo",
          "command" : "Commandline: apt-get install zmap",
          "data_type" : "apt:history:line",
          "display_name" : "GZIP:/var/log/apt/history.log.1.gz",
          "filename" : "/var/log/apt/history.log.1.gz",
          "packages" : "Install: zmap:amd64 (1.1.0-1)",
          "parser" : "apt_history",
          "tag" : [
            "sigma_lnx_susp_zenmap"
          ]
        }
      }
    ]
  }
}

OR

curl -X GET "tselastic:9200/dd627756168a47869e7d5022216b1906/_search?pretty" -H 'Content-Type: application/json' -d'
> {
>   "query": {
>     "query_string": {
>       "query": "message.keyword:/.*apt\\-get install zmap.*/",
>       "default_field": "message"
>     }
>   }
> }
> '
{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "dd627756168a47869e7d5022216b1906",
        "_type" : "_doc",
        "_id" : "iGuujnUBnpFBqXiee6Kr",
        "_score" : 1.0,
        "_source" : {
          "message" : "Install: zmap:amd64 (1.1.0-1) [Commandline: apt-get install zmap]",
          "timestamp" : 123456791,
          "datetime" : "2015-07-24T19:01:03+00:00",
          "timestamp_desc" : "foo",
          "command" : "Commandline: apt-get install zmap",
          "data_type" : "apt:history:line",
          "display_name" : "GZIP:/var/log/apt/history.log.1.gz",
          "filename" : "/var/log/apt/history.log.1.gz",
          "packages" : "Install: zmap:amd64 (1.1.0-1)",
          "parser" : "apt_history",
          "tag" : [
            "sigma_lnx_susp_zenmap"
          ]
        }
      }
    ]
  }
}
-----------------lnx3_susp_zenmap.yml REGEX VERSION-----------------------
title: Suspicious Installation of Zenmap
id: 5266a592-b793-11ea-b3de-0242ac130005
description: Detects suspicious installation of Zenmap
references:
    - https://rmusser.net/docs/ATT&CK-Stuff/ATT&CK/Discovery.html
author: Alexander Jaeger
date: 2020/06/26
modified: 2020/06/26
logsource:
    product: linux
    service: shell
detection:
    keywords:
        # Generic suspicious commands
        - message.keyword|re: ".*apt\\-[a-z]{3} install zmap.*"
    condition: keywords
falsepositives:
    - Unknown
level: high
-------------------------------------------------------------------------
-----------------In timesketch CELERY LOG-------------------
[2020-11-04 10:00:28,828] celery.pool/DEBUG TaskPool: Apply <function _fast_trace_task at 0x7f2dc8972f28> (args:('timesketch.lib.tasks.run_sketch_analyzer', 'ccba2f83-4018-4317-b9f2-0a5b055bc12f', {'lang': 'py', 
'task': 'timesketch.lib.tasks.run_sketch_analyzer', 'id': 'ccba2f83-4018-4317-b9f2-0a5b055bc12f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '
c6d9c751-3871-46fc-ab93-8a685286b11a', 'parent_id': 'c6d9c751-3871-46fc-ab93-8a685286b11a', 'argsrepr': "('dd627756168a47869e7d5022216b1906', 1, 20, 'sigma')", 'kwargsrepr': '{}', 'origin': 'gen155@26fa720cf417',
 'reply_to': '3f4da99c-029f-340f-aa69-c5f4ecea63f2', 'correlation_id': 'ccba2f83-4018-4317-b9f2-0a5b055bc12f', 'hostname': 'celery@26fa720cf417', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priori
ty': 0, 'redelivered': None}, 'args': ['dd627756168a47869e7d5022216b1906', 1, 20, 'sigma'], 'kwargs': {}}, b'[["dd627756168a47869e7d5022216b1906", 1, 20, "sigma"], {}, {"callbacks": null, "errbacks": null, "chain
": [], "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2020-11-04 10:00:28,831] celery.worker.request/DEBUG Task accepted: timesketch.lib.tasks.run_sketch_analyzer[ccba2f83-4018-4317-b9f2-0a5b055bc12f] pid:155
[2020-11-04 10:00:28,835] timesketch.analyzers.sigma_tagger/DEBUG [sigma] Loading config from /etc/timesketch/sigma_config.yaml
[2020-11-04 10:00:28,870] timesketch.analyzers.sigma_tagger/INFO [sigma] Reading rules from /usr/local/src/timesketch/data/sigma/rules/lnx3_susp_zenmap.yml
[2020-11-04 10:00:28,878] timesketch.analyzers.sigma_tagger/INFO [sigma] Generated query (data_type:("shell\:zsh\:history" OR "bash\:history\:command" OR "apt\:history\:line" OR "selinux\:line") AND message:/.*ap
t\-[a-z]{3} install zmap.*/)
[2020-11-04 10:00:28,878] urllib3.connectionpool/DEBUG Starting new HTTP connection (1): tselastic:9200
[2020-11-04 10:00:28,884] urllib3.connectionpool/DEBUG http://tselastic:9200 "POST /dd627756168a47869e7d5022216b1906/_refresh HTTP/1.1" 200 49
[2020-11-04 10:00:28,887] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 10:00:28,901] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /dd627756168a47869e7d5022216b1906/_search?_source_includes=human_readable%2Ctag%2Cmessage%2C__ts_emojis&scroll=1m&search_type=quer
y_then_fetch HTTP/1.1" 200 297
[2020-11-04 10:00:28,902] timesketch.analyzers.sigma_tagger/INFO [sigma] Reading rules from /usr/local/src/timesketch/data/sigma/rules/linux/lnx_susp_zenmap.yml
[2020-11-04 10:00:28,907] timesketch.analyzers.sigma_tagger/INFO [sigma] Generated query (data_type:("shell\:zsh\:history" OR "bash\:history\:command" OR "apt\:history\:line" OR "selinux\:line") AND "*apt\-get\ i
nstall\ zmap*")
[2020-11-04 10:00:28,909] urllib3.connectionpool/DEBUG http://tselastic:9200 "POST /dd627756168a47869e7d5022216b1906/_refresh HTTP/1.1" 200 49
[2020-11-04 10:00:28,912] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 10:00:28,924] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /dd627756168a47869e7d5022216b1906/_search?_source_includes=human_readable%2Ctag%2Cmessage%2C__ts_emojis&scroll=1m&search_type=quer
y_then_fetch HTTP/1.1" 200 544
[2020-11-04 10:00:28,926] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET / HTTP/1.1" 200 541
[2020-11-04 10:00:28,935] urllib3.connectionpool/DEBUG http://tselastic:9200 "GET /_search/scroll?scroll=5m HTTP/1.1" 200 296
[2020-11-04 10:00:29,051] urllib3.connectionpool/DEBUG http://tselastic:9200 "POST /_bulk HTTP/1.1" 200 258
[2020-11-04 10:00:29,052] timesketch.tasks/INFO [sigma] result: Applied 1 tags
* lnx3_susp_zenmap: 0
* lnx_susp_zenmap: 1

[2020-11-04 10:00:29,053] celery.app.trace/INFO Task timesketch.lib.tasks.run_sketch_analyzer[ccba2f83-4018-4317-b9f2-0a5b055bc12f] succeeded in 0.22285113390535116s: 'dd627756168a47869e7d5022216b1906'

If you remove line 124 (https://github.com/google/timesketch/blob/cdf0275dc0314e14385797c68a8c774f50be3344/timesketch/lib/analyzers/sigma_tagger.py#L124), it works fine (regex apply on raw string and not splitted string)! (I dont understand problem in issue https://github.com/google/timesketch/issues/1199#issuecomment-639475885 -> no example rule).

I asked access to slack, thank.

[lprat]

Hi,

I found problem for tag on same event: https://github.com/google/timesketch/blob/8f2e48713ba3f2b060440c1fde6436b182fcb84a/timesketch/lib/datastores/elastic.py#L672 . If you add 'flush_interval=1' in https://github.com/google/timesketch/blob/c2437c6b781fd7857669819c7c0761b59a4776b4/timesketch/lib/analyzers/interface.py#L163-L165 then tag written... For resolv problem, you should verify if 'id' event already present in "self.import_events" and if present then merge.

You have same problem on some analyzers (ex: https://github.com/google/timesketch/blob/master/timesketch/lib/analyzers/feature_extraction.py#L234).

Thank

[jaegeral]

Ok will have a look at it.

[jaegeral]

Confirmed issue.

[jaegeral]

Ok this one should be fixed with the linked PR

[lprat]

Hi,

Sorry i reopen because fix only one problem (tag), but not ".keyword" problem in sigma: https://github.com/google/timesketch/issues/1450#issuecomment-721615767

Thanks for help.

Lionel

[jaegeral]

Hi,

Sorry i reopen because fix only one problem (tag), but not ".keyword" problem in sigma: #1450 (comment)

Thanks for help.

Lionel

I opened a new issue to track, having multiple issues in one is very hard to keep on top.