[Rule Tuning] Endpoint Rules Broken for Endgame Events from Pull #2593

[PoppaShell]

Link to rule

Pretty much all rules in the commit for Pull Request #2593 https://github.com/elastic/detection-rules/commit/59da2da4740da52df15ce11139150b61cd57f9e1

Description

When Pull Request #2593 was committed, it added host information to the queries by using the field name of "host.os.type" to isolate each endpoint query to the appropriate host type of "windows", "linux" or "macos". I think this is a good idea overall. But it broke all Endpoint Rules for Endgame Events. This is because the Endgame Stream Integration doesn't send information for the field "host.os.type". It has broken all of our alerts we have enabled for the "Endgame Elastic" tagged rules.

I would expect that changes to Detection Rules would take into consideration both the Elastic Agent & Endgame Agent Data. And with that in mind, I can think of two viable options that would not have broken this flow and/or would fix the issue for the future.

1. Update the Elastic Stream Integration to include the "host.os.type" field instead of or in addition to "observer.os.type"
2. Change the endpoint rules to look for "host.os.type" and "observer.os.type" (ex: process where host.os.type == "windows" or observer.os.type == "windows")

Screenshots You can see in this screenshot that when we pushed the Pre-Built Rules update on 04/24 that we stopped getting alerts.

Example Data

You can see in the commit (https://github.com/elastic/detection-rules/commit/59da2da4740da52df15ce11139150b61cd57f9e1) that this "host.os.type" was added to all of the endpoint rules. And that was the whole goal of Pull Request #2593. But if you look at the events being sent from Endgame (sample below) that this field is not present.

{
  "@timestamp": 1683151638000,
  "agent": {
    "id": "<redacted>",
    "version": "3.62.5",
    "name": "Endgame",
    "type": "Endgame"
  },
  "data_stream": {
    "dataset": "endgame.process",
    "namespace": "endgame",
    "type": "logs"
  },
  "ecs": {
    "version": "8.4.0"
  },
  "event": {
    "kind": "event",
    "category": "process",
    "dataset": "endgame.process",
    "type": "start",
    "action": "creation_event",
    "module": "endgame",
    "sequence": 2055382810
  },
  "host": "<redacted>",
  "observer": {
    "hostname": "<redacted>",
    "ip": "<redacted>",
    "product": "SMP",
    "vendor": "Endgame",
    "version": "3.29.2",
    "os": {
      "family": "windows",
      "name": "Windows",
      "version": "10.0",
      "platform": "windows",
      "type": "windows"
    }
  },
  "organization": {
    "id": "<redacted>",
    "name": "Elastic Endgame"
  },
  "user": {
    "domain": "NT AUTHORITY",
    "full_name": "SYSTEM",
    "id": "S-1-5-18",
    "name": "SYSTEM"
  },
  "process": {
    "args": [
      "C:\\WINDOWS\\system32\\cmd.exe",
      "/c",
      "ver"
    ],
    "args_count": 3,
    "command_line": "C:\\WINDOWS\\system32\\cmd.exe /c \"ver\"",
    "entity_id": "2a0dadb7b478f89a2e482838bc7d1240574ed60532be24330a5fe904fd2d0756",
    "executable": "C:\\Windows\\System32\\cmd.exe",
    "name": "cmd.exe",
    "title": "cmd.exe",
    "pid": 9024,
    "code_signature": {
      "status": "trusted",
      "subject_name": "Microsoft Windows",
      "exists": true,
      "trusted": true,
      "valid": true
    },
    "hash": {
      "md5": "<redacted>",
      "sha1": "<redacted>",
      "sha256": "<redacted>"
    },
    "pe": {
      "original_file_name": "Cmd.Exe"
    },
    "parent": {
      "pid": 2680,
      "name": "<redacted>",
      "title": "<redacted>",
      "executable": "C:\\Program Files (x86)\\<redacted>",
      "args": [
        "C:\\Program Files (x86)\\<redacted>",
        "<redacted>"
      ],
      "args_count": 2,
      "command_line": "\"C:\\Program Files (x86)\\<redacted>"
    }
  },
  "related": {
    "hash": [
      "<redacted>",
      "<redacted>",
      "<redacted>"
    ],
    "hosts": [
      "<redacted>",
      "NT AUTHORITY"
    ],
    "ip": [
      "<redacted>"
    ],
    "user": [
      "SYSTEM"
    ]
  },
  "endgame": {
    "event_subtype_full": "creation_event",
    "event_type_full": "process_event",
    "opcode": 1,
    "pid": 9024,
    "process_name": "cmd.exe",
    "process_path": "C:\\Windows\\System32\\cmd.exe",
    "serial_event_id": 2055382810,
    "unique_pid": 2055382810,
    "user_domain": "NT AUTHORITY",
    "user_name": "SYSTEM",
    "user_sid": "S-1-5-18",
    "ppid": 2680,
    "command_line": "C:\\WINDOWS\\system32\\cmd.exe /c \"ver\"",
    "parent_command_line": "\"C:\\Program Files (x86)\\<redacted>",
    "parent_process_name": "<redacted>",
    "parent_process_path": "C:\\Program Files (x86)\\<redacted>",
    "md5": "<redacted>",
    "sha1": "<redacted>",
    "sha256": "<redacted>",
    "unique_ppid": 2055377456,
    "signature_signer": "Microsoft Windows",
    "signature_status": "trusted",
    "original_file_name": "Cmd.Exe",
    "authentication_id": 999,
    "elevated": true,
    "integrity_level": "system",
    "elevation_type": "default",
    "timestamp_utc": "2023-05-03 22:07:18Z",
    "timestamp": 133276252380621920
  },
  "_time": 1683151638
}

[PoppaShell]

Additionally, I would highly recommend actually keeping the Pre-Built Rules Change Log & Documentation updated to reflect what is actually in the Rules Repo that gets pushed to Elastic Cloud. When Elastic Security notifies you in the GUI that there are new Pre-Built Rule Updates, it references the Change Log linked below. None of the Rules listed have the updated from Pull Request #2593 listed. So I have no idea or heads up this field would be added so I could catch that Endgame doesn't sent the appropriate date. But I shouldn't even have to worry about doing that since Elastic owns Endgame & the Stream Integration to Elastic and it's corresponding rules.

https://www.elastic.co/guide/en/security/current/prebuilt-rules-changelog.html

Until this is fixed, I will be adding the needed field using Cribl, as we have all Endgame routing through there. But for those that don't have something in-between, this is a solid inconvenience. I've lost 9 dates of potential alerts. I should have caught this earlier, but it wasn't the most straight forward find.

[w0rk3r]

Hey @PoppaShell, while developing the PR, we have tested against an Endgame setup (example events can be found here and here), but I'm not sure if any specifics on your deployment could cause this field not to be populated

[PoppaShell]

@w0rk3r I shared a sample windows log coming directly from Endgame SMP, hosted by Endgame in the cloud. And I can share as many as you need me to. And below is a Linux sample. But I also just noticed something else new. There is not a "data_stream" object. Did Endgame now switch to using Data Streams recently? And maybe that is the issue? We have it sending to Cribl to do some reduction and then forwarding it on with the Bulk API, just as Endgame normally would to send it to Elastic. Is it now natively sending to a Data Stream and then doing other post processing on the Elastic side to add the "host.os.type" field?

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[PoppaShell]

This is stale because my last comment was ignored, apparently. But I would have loved to get more dialogue on this issue. My assumption is that Endgame's Elastic Stream Integration switched to using Datastreams and Ingest Pipelines to add this field value upon ingest. But I was able to verify it is not included in the logs coming straight from Endgame.

[PoppaShell]

Reopening, as I mistakenly clicked the close button.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.