[Security Solution] No geo data from Microsoft 365 Integration

[tandemkid]

**Describe the bug: Geo IP information is not being generated from the M365 integration.

**Kibana/Elasticsearch Stack version: 8.5.2

**Server OS version: Elastic Cloud

**Browser and Browser OS versions: Chrome 107.0.5304.121

**Elastic Endpoint version: 8.5.2

**Original install method (e.g. download page, yum, from source, etc.): Elastic Agent from the downloads page via Curl.

**Functional Area (e.g. Endpoint management, timelines, resolver, etc.): ingest pipelines

Steps to reproduce:

1. Install the Microsoft 365 Integration
2. Ingest traffic
3. Review documents and there is no geo IP data.

**Current behavior: Two keyword fields are present ClientIP and ActorIPAddress. Neither are IP types and no geo data is presented.

**Expected behavior: Geo IP information is presented in the document.

**Screenshots (if relevant):

**Any additional context (logs, chat logs, magical formulas, etc.): I think this is because of how the ClientIP and ActorIPAddress fields type is keyword.

{ "_index": ".ds-logs-o365.audit-removed-2022.12.03-000089", "_id": "c15b3f18-69b1-48eb-bbfd-6b2b8e545b02", "_version": 1, "_score": 0, "_source": { "input": { "type": "o365audit" }, "agent": { "name": "fk-02yp-ea", "id": "59122787-c708-406b-8112-459123afedd1", "type": "filebeat", "ephemeral_id": "5164dc0c-13c9-4857-bd32-fdf3ca042be9", "version": "8.5.2" }, "@timestamp": "2022-12-03T01:04:40.000Z", "ecs": { "version": "8.0.0" }, "data_stream": { "namespace": "removed", "type": "logs", "dataset": "o365.audit" }, "elastic_agent": { "id": "removed-c708-406b-8112-459123afedd1", "version": "8.5.2", "snapshot": false }, "_conf": { "tenants": { "removed-e289-4e18-8f6d-0f806c781eca": "removed.onmicrosoft.com" } }, "event": { "agent_id_status": "verified", "ingested": "2022-12-03T01:09:20Z", "dataset": "o365.audit" }, "o365audit": { "AzureActiveDirectoryEventType": 1, "UserKey": "removed-6b56-48e3-91ba-7ee4860feba2", "ActorIpAddress": "removed.133.245.3", "Operation": "UserLoggedIn", "OrganizationId": "removed-e289-4e18-8f6d-0f806c781eca", "ExtendedProperties": [ { "Value": "Success", "Name": "ResultStatusDetail" }, { "Value": "16", "Name": "UserAuthenticationMethod" }, { "Value": "OAuth2:Token", "Name": "RequestType" } ], "IntraSystemId": "removed-69b1-48eb-bbfd-6b2b8e545b02", "Target": [ { "Type": 0, "ID": "00000002-0000-0000-c000-000000000000" } ], "RecordType": 15, "Version": 1, "ModifiedProperties": [], "SupportTicketId": "", "Actor": [ { "Type": 0, "ID": "removed-6b56-48e3-91ba-7ee4860feba2" }, { "Type": 5, "ID": "Sync_AZURE-CONNECT_removed@removed.onmicrosoft.com" } ], "DeviceProperties": [ { "Value": "Windows 8", "Name": "OS" }, { "Value": "Other", "Name": "BrowserType" }, { "Value": "False", "Name": "IsCompliantAndManaged" }, { "Value": "removed-7d93-456c-a788-36246ece4b6c", "Name": "SessionId" } ], "ActorContextId": "removed-e289-4e18-8f6d-0f806c781eca", "ResultStatus": "Success", "ObjectId": "00000002-0000-0000-c000-000000000000", "ErrorNumber": "0", "ClientIP": "removed.133.245.3", "Workload": "AzureActiveDirectory", "UserId": "Sync_AZURE-CONNECT_removed@removed.onmicrosoft.com", "TargetContextId": "removed-e289-4e18-8f6d-0f806c781eca", "CreationTime": "2022-12-03T01:04:40", "Id": "removed-69b1-48eb-bbfd-6b2b8e545b02", "InterSystemsId": "removed-deed-415a-b86d-c13d81178c64", "ApplicationId": "removed-e479-49de-ae31-7812af012ed8", "UserType": 0 }, "tags": [ "forwarded", "o365-audit" ] }, "fields": { "elastic_agent.version": [ "8.5.2" ], "o365audit.ObjectId": [ "00000002-0000-0000-c000-000000000000" ], "o365audit.RecordType": [ 15 ], "o365audit.DeviceProperties.Value": [ "Windows 8", "Other", "False", "b642ddff-7d93-456c-a788-36246ece4b6c" ], "o365audit.TargetContextId": [ "ad8e5256-e289-4e18-8f6d-0f806c781eca" ], "o365audit.DeviceProperties.Name": [ "OS", "BrowserType", "IsCompliantAndManaged", "SessionId" ], "agent.name": [ "fk-02yp-ea" ], "event.agent_id_status": [ "verified" ], "o365audit.Workload": [ "AzureActiveDirectory" ], "o365audit.SupportTicketId": [ "" ], "o365audit.ApplicationId": [ "removed-e479-49de-ae31-7812af012ed8" ], "input.type": [ "o365audit" ], "data_stream.type": [ "logs" ], "tags": [ "forwarded", "o365-audit" ], "o365audit.ActorIpAddress": [ "removed.133.245.3" ], "agent.id": [ "59122787-c708-406b-8112-459123afedd1" ], "o365audit.AzureActiveDirectoryEventType": [ 1 ], "ecs.version": [ "8.0.0" ], "agent.version": [ "8.5.2" ], "o365audit.IntraSystemId": [ "removed-69b1-48eb-bbfd-6b2b8e545b02" ], "o365audit.Target.ID": [ "00000002-0000-0000-c000-000000000000" ], "o365audit.ExtendedProperties.Name": [ "ResultStatusDetail", "UserAuthenticationMethod", "RequestType" ], "o365audit.ResultStatus": [ "Success" ], "o365audit.UserId": [ "Sync_AZURE-CONNECT_removed@removed.onmicrosoft.com" ], "o365audit.ClientIP": [ "removed.133.245.3" ], "_conf.tenants.ad8e5256-e289-4e18-8f6d-0f806c781eca": [ "removed.onmicrosoft.com" ], "agent.type": [ "filebeat" ], "o365audit.ExtendedProperties.Value": [ "Success", "16", "OAuth2:Token" ], "event.module": [ "o365" ], "o365audit.Operation": [ "UserLoggedIn" ], "o365audit.ErrorNumber": [ "0" ], "o365audit.Actor.ID": [ "removed-6b56-48e3-91ba-7ee4860feba2", "Sync_AZURE-CONNECT_removed@removed.onmicrosoft.com" ], "elastic_agent.snapshot": [ false ], "o365audit.CreationTime": [ "2022-12-03T01:04:40" ], "o365audit.InterSystemsId": [ "8cba189c-deed-415a-b86d-c13d81178c64" ], "elastic_agent.id": [ "59122787-c708-406b-8112-459123afedd1" ], "data_stream.namespace": [ "removed" ], "o365audit.UserType": [ 0 ], "o365audit.Actor.Type": [ 0, 5 ], "event.ingested": [ "2022-12-03T01:09:20.000Z" ], "@timestamp": [ "2022-12-03T01:04:40.000Z" ], "o365audit.ActorContextId": [ "ad8e5256-e289-4e18-8f6d-0f806c781eca" ], "o365audit.OrganizationId": [ "ad8e5256-e289-4e18-8f6d-0f806c781eca" ], "data_stream.dataset": [ "o365.audit" ], "o365audit.Version": [ 1 ], "agent.ephemeral_id": [ "5164dc0c-13c9-4857-bd32-fdf3ca042be9" ], "o365audit.Id": [ "c15b3f18-69b1-48eb-bbfd-6b2b8e545b02" ], "o365audit.Target.Type": [ 0 ], "o365audit.UserKey": [ "8adb0617-6b56-48e3-91ba-7ee4860feba2" ], "event.dataset": [ "o365.audit" ] } }

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[ebeahan]

@tandemkid the o365 integration is maintained here in the integration repo, so the issue has been transferred over from Kibana.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[efd6]

@tandemkid How are you ingesting this?

I would expect client.ip and source.ip to be present in the document (the ingest pipeline uses the source.ip field as the input for the GeoIP query and it gets this field indirectly from the o365audit.ClientIPAddress, o365audit.ClientIP or o365audit.ActorIpAddress fields) and for the event field group to be holding more data than what it does (at least event.code should be set to "AzureActiveDirectoryStsLogon") but the full event group is:

        "event": {
            "agent_id_status": "verified",
            "ingested": "2022-12-03T01:09:20Z",
            "dataset": "o365.audit"
        },

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!