RQL query not working with "raw_log"

[greycel]

Hi Team,

I've got an event with the below data, trying to write an RQL query to auto-dismiss events but wasn't able to get the query working.

//Sample Event
{
  host:{
    name: "DESKTOP-D7M89F8"
  },
  event_host: "DESKTOP-D7M89F8",
  process.executable: "c:\windows\system32\svchost.exe",
  process:{
    name:"svchost.exe"
  },
  event.code: 22,
  winlog.event_data.queryname: "wpad",
  rule:{
  name: "Spoofed Windows process"
  description: "Observed Spoofed Windows process",
  id: "10f91d1d-96c5-4b99-af99-682dacc798f3"
  },
  priority: 8,
  user:{
    name: ""
  },
  event:{
  kind: "alert",
  created: "2023-08-23T04:42:15.059Z"
  }
} 

The below query matches the event with the process name svchost.exe using raw_log event field:

//Query matches
title = "Spoofed Windows process"
and raw_log.process.name eq "svchost.exe"

But the query fails to match the value wpad in the field winlog.event_data.queryname

//Query doesn't match
title = "Spoofed Windows process"
and raw_log.winlog.event_data.queryname eq "wpad"

//Query doesn't match
title = "Spoofed Windows process"
and raw_log.process.executable eq "c:\\windows\\system32\\svchost.exe"

When checked for fields priority: 8, event.code: 22 Query matches for the field priority, but fails to match with field raw_log.event.code

//Query matches
title = "Spoofed Windows process"
and raw_log.priority eq 8

//Query doesn't match
title = "Spoofed Windows process"
and raw_log.event.code eq 22

Not sure if the queries are getting failed due to the presence of dot in the event field name, could you pls advise on how I can get this working.

winlog.event_data.queryname
event.code

[n3tsurge]

Are you by chance using Elasticsearch as your alert source?

[greycel]

Using OpenSearch with Elastalert for the rule engine

[n3tsurge]

Gotcha, the reason I ask is Elastic had switched to using flattened keys instead of actual nested JSON.

This is bad

{"field.name.here": "value"}

This is good

{ "field": {"name": {"here": "value"}}}

This behavior ended up breaking how we do field extraction. It has been fixed in the Agent side but not in RQL yet. I will work it in to the next release which should be coming really soon.

The work around would be to not index with flat keys. It appears you have both flat keys and normal nested keys in your example event.

[n3tsurge]

Which build of ReflexSOAR are you currently running? The main stream release v2022.06.00 or one of the dev builds?

[greycel]

Using the latest version v2023.08.07-rc0

[n3tsurge]

Oh perfect, you should have this fix sooner than most people then, that's the build we run internally we have some other items staged to get moved into a new dev release client on Monday next week, this bug fix included.

It has some nice features in it like SAML authentication, multi-key threshold rules, dead source detection rules, a bunch of bug fixes, notification channels, the start of our integrations/actions framework, standardizing the markdown editors, UI improvements, etc.

I am curious if you are running the latest RC have you tried Detections instead of ElastAlert?

[greycel]

Thanks for the update, you guys are doing great really appreciate it...

awaiting for the new release. Nope, Haven't tested detections yet, but will try it soon. Also, documentation would come in handy for these new features.