[Filebeat] [Azure module] - Improve ECS utilization

[defendable-forfot]

We are ingesting Azure data (signinlogs, activitylogs, and auditlogs) into our Elasticsearch for search, detection in Elastic Security and visualization through Kibana. However, we have noticed a few specific fields where the Azure module does not optimally utilize ECS.

Note: we are running filebeat version 8.1.3, but have noticed that none of the newer releases solves our issues.

Signinlogs

azure.tenant_id

    ECS fields:  cloud.account.id

    Suggestion: Populate the cloud.account.id field with data from the azure.tenant_id field.

Activitylogs

azure.tenant_id

    ECS fields:  cloud.account.id

    Suggestion: Populate the cloud.account.id field with data from the azure.tenant_id field.

source.ip | related.ip | host.ip

    ECS fields: source.address | source.ip | related.ip | host.ip

    Suggestion: The IP address related to the event is currently populated in the source.ip, host.ip and related.ip fields, however it should also populate the source.address field.

azure.activitylogs.category

    ECS fields: event.category

    Suggestion: Events in the under activitylogs will always be related to ‘'configuration’' type changes. We recommend that the value in event.category is set to ‘'configuration’'.

Auditlogs

azure.tenant_id

    ECS fields:  cloud.account.id

    Suggestion: Populate the cloud.account.id field with data from the azure.tenant_id field.

azure.auditlogs.properties.initiated_by.user.ipAddress

    ECS fields: client.ip | source.address | source.ip | related.ip

    Suggestion: The source.address field is not populated with the IP related to the event. Other ECS fields are set and we recommend that source.address is also populated.

azure.auditlogs.properties.initiated_by.user.userPrincipalName

    ECS fields: user.email | user.domain | user.name | client.user | related.user

    Suggestion: The userPrincipalName field contains the email address of the user that has performed the action. Subsequently the data in this field can be used to populate the following ECS fields: user.email, user.domain, user.name, client.user and related.user. Note: the data should be parsed to extract relevant data to the fields, not simply copied in it’s raw format. 

azure.auditlogs.properties.target_resources.*.user_principal_name

    ECS fields: related.user

    Suggestion: The targeted user of the event should also be populated in the related.user field. This does not currently happen.

azure.auditlogs.properties.additional_details.[*].key | azure.auditlogs.properties.additional_details.[*].value

    ECS fields: user_agent.original | user_agent.name | user_agent.version

    Suggestion: If a .key within .addtional_details is set to ‘'User-Agent’' the corresponding .value can be extracted and populate the user_agent fields.

azure.auditlogs.properties.category

    ECS fields: event.category

    Suggestion: In cases where the azure.auditlogs.properties.category field is set to RoleManagement it can be set to ‘'iam’' under event.category. For cases where azure.auditlogs.properties.category is set to ApplicationManagement and Policy, event.category can be set to ‘'configuration’'.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

Can you clarify the source.id field you mention? This is not an ECS field and I cannot find it in the packages field definitions.

[defendable-forfot]

Reviewed it once more and it seems the source.id field may have been referenced by mistake, under the assumption that such a field existed. As you mention, it is not an ECS field. Based on existing ECS fields it is added to an appropriate field.

[efd6]

Would you please edit the issue to update with the new information.

[defendable-forfot]

Updated and removed initial request related to source.id

[efd6]

Can I confirm that you are using filebeat directly rather than via the azure integration package and elastic agent?

[defendable-forfot]

Yes, we are using Filebeat directly and not through Elastic Agent.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!