Open jvalente-salemstate opened 4 months ago
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
It fails at:
{
"processor\_type": "json",
"status": "error",
"if": {
"condition": "ctx.o365audit?.containsKey('Data') == true",
"result": true
},
"error": {
"root\_cause": \[
{
"type": "x\_content\_parse\_exception",
A possible workaround to make the json
decoding work would be to update the ingest pipeline logs-o365.audit-...
with:
{
"gsub": {
"field": "o365audit.Data",
"pattern": "\\"QueryTime\\":\\"\[0-9/\]+ \[0-9:\]+ \[AP\]M\\",",
"replacement": "",
"if": "ctx.o365audit?.containsKey('Data') == true && ctx.o365audit?.RecordType == '64'"
}
},
Just before:
...
{
"json": {
"field": "o365audit.Data",
"if": "ctx.o365audit?.containsKey('Data') == true"
}
},
...
With the gsub
processor, we're patching any occurrence of the QueryTime
which uses the non-ISO8601 format.
We can execute it only on the RecordType == 64
.
This looks like it was introduced in v2.0.0 or v2.1.0 as this only began showing in mid January. See #8571 and #8803 for the changes in the specific PR.
Almost entirely, this occurs to
o365.audit.record.type: 64
, for Automated Investigation & Response (AIR) events. When the pipeline fails, it does not continue to rename fields nested undero365audit
too365.audit
and these events will not match under any queries for those fields.Many events do end up parsing correctly, but I've got roundly 9200 since January 22nd with an
error.message
that initially contained:At some point after, this became:
Formatting edited for readability
The AIR events contain an array of entities related to an alert. An alert might have multiple instances of one entity type and it seems like most,if not all of these, are when the alert has more than one mail cluster that may be sender+IP+subject, sender+attachments, etc (with the ID being analogous for the
fingerprint
processor in elastic, based on those values). Since something likeQueryTime
has exists in each, the above error is thrown.Sample data: