[filebeat][o365] Mapping problem on o365.audit.Parameters

[willemdh]

• Version: 7.9.2
• Operating System: CentOS 8
• Discuss Forum URL: https://discuss.elastic.co/t/ingesting-o365-logs-but-having-issues-with-specific-mappings/256081
• Steps to Reproduce: Ingest lots of o365 audit logs some with o365.audit.Parameters and some without o365.audit.Parameters

Error:

Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [o365.audit.Parameters] tried to parse field [Parameters] as object, but found a concrete value"}

Workaround:

- drop_fields:
    when:
      equals:
        event.module: 'o365'
    fields: ["o365.audit.Parameters"]

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[cakarlen]

Bump on this issue as I experienced this exact error as OP described above when turning up o365 logging. Verified my Azure app has the correct permissions and manually tested API creds to verify

[willemdh]

@cakarlen The workaround works fine for me, but we shouldnt have to do this..

[P1llus]

Some quick updates on this, sorry for the silence here. The 7.13.2 filebeat module has an update for this: https://github.com/elastic/beats/pull/26164

For the package(fleet version), it will come out with a requirement of 7.14, as the package has some features that it is required for. This can be followed here: https://github.com/elastic/integrations/pull/1136

The current implementation will copy the Parameters field to Parameters._raw if its a string. This is to make sure that ingestion works. A later version we might want to come back and build more parsers for these string fields, but the current issue is that the string values in the parameters field does not follow a specific format, and varies quite a lot, so this will have to be the initial change.

[kh31r0n]

• Version: 7.15.2

• Running on: kubernetes Error: object mapping for [o365audit.ModifiedProperties] tried to parse field [null] as object, but found a concrete value Workaround doesn't work

  Same problem but with different field.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[willemdh]

+1

[willemdh]

Multiple interesting fields are also unmapped and hence unusable for filtering / aggregating etc;.

For example

o365.audit.ExtendedProperties.RequestType o365.audit.ExtendedProperties.ResultStatusDetail o365.audit.ExtendedProperties.UserAuthenticationMethod

[novaksam]

Also having this issue, with o365.audit.Platform It appears that some of the most common events have a platform of '1', triggering numeric detection, and breaking future events.

[novaksam]

I'm using the following logstash filter to prune/handle some of this

  if [o365] {
    if [o365][audit][platform] == "1" {
      mutate {
        replace => { "[o365][audit][platform]" => "one"}
      }
    }

    # "reason"=>"mapper [o365.audit.FileSizeBytes] cannot be changed from type [long] to [float]"}}}
    if [o365][audit][FileSizeBytes] =~ \.\d*$ {
      mutate {
        convert => {
          "[o365][audit][FileSizeBytes]" => "integer"
        }
      }
    }
  }`

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)