Add a related.url field to facilitate searching for URLs that appear in various other fields of an event.
Motivation:
This was requested by a user in order to improve mappings for data sources that have multiple URL fields, such as data from the o365 integration.
The closest existing field is related.hosts, which is for "All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases."
The addition of related.domain was suggested as an alternative to related.url. I have focused here on related.url as it is more distinct from the related.hosts use case.
Detailed Design:
A related.url field could be populated with the same kind of values as url.full when possible, or the same kind as url.original if that is the most complete value available.
Setting a field type of wildcard would match the url.full and url.original fields. A .text multi-field could be added.
Summary
Add a
related.urlfield to facilitate searching for URLs that appear in various other fields of an event.Motivation:
This was requested by a user in order to improve mappings for data sources that have multiple URL fields, such as data from the o365 integration.
The closest existing field is
related.hosts, which is for "All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases."The addition of
related.domainwas suggested as an alternative torelated.url. I have focused here onrelated.urlas it is more distinct from therelated.hostsuse case.Detailed Design:
A
related.urlfield could be populated with the same kind of values asurl.fullwhen possible, or the same kind asurl.originalif that is the most complete value available.Setting a field type of
wildcardwould match theurl.fullandurl.originalfields. A.textmulti-field could be added.Examples from o365 integration - not the most compelling, but this is what was readily available in test data
```json { "@timestamp": "2020-02-14T19:00:00.000Z", "ecs": { "version": "8.11.0" }, "event": { "action": "AlertEntityGenerated", "category": [ "web" ], "code": "SecurityComplianceAlerts", "id": "448854d7-81f6-4a06-d31a-08d7b1c1fb2f", "kind": "alert", "outcome": "success", "provider": "SecurityComplianceCenter", "type": [ "info" ] }, "host": { "id": "b86ab9d4-fcf1-4b11-8a06-7a8f91b47fbd", "name": "mytenant.onmicrosoft.com" }, "message": "New alert", "o365": { "audit": { "AlertId": "5ba6e029-8b6e-13bd-b800-08d7b180173c", "AlertType": "System", "CreationTime": "2020-02-14T19:00:00", "Data": { "eid": "asr@testsiem.onmicrosoft.com", "etype": "User", "flattened": { "eid": "asr@testsiem.onmicrosoft.com", "etype": "User", "lon": "GrantAdminPermission", "op": "GrantAdminPermission", "suid": "asr@testsiem.onmicrosoft.com", "tdc": "1", "te": "2020-02-14T18:54:45.0000000Z", "tid": "b86ab9d4-fcf1-4b11-8a06-7a8f91b47fbd", "ts": "2020-02-14T18:54:45.0000000Z", "ut": "Admin" }, "lon": "GrantAdminPermission", "op": "GrantAdminPermission", "suid": "asr@testsiem.onmicrosoft.com", "tdc": "1", "te": "2020-02-14T18:54:45.000Z", "tid": "b86ab9d4-fcf1-4b11-8a06-7a8f91b47fbd", "ts": "2020-02-14T18:54:45.000Z", "ut": "Admin" }, "ObjectId": "asr@testsiem.onmicrosoft.com", "RecordType": "40", "ResultStatus": "Succeeded", "Severity": "Low", "Source": "Office 365 Security & Compliance", "Status": "Active", "UserId": "SecurityComplianceAlerts", "UserKey": "SecurityComplianceAlerts", "UserType": "4", "Version": "1" } }, "organization": { "id": "b86ab9d4-fcf1-4b11-8a06-7a8f91b47fbd", "name": "mytenant.onmicrosoft.com" }, "rule": { "category": "AccessGovernance", "description": "asr@testsiem.onmicrosoft.com", "id": "17d51759-88e1-40c1-8df3-20bcf2e43057", "name": "Elevation of Exchange admin privilege", "reference": [ "http://example.net/alert", // URL 1 "http://example.net/info" // URL 2 ], "ruleset": "User" }, "tags": [ "preserve_original_event" ], "user": { "id": "SecurityComplianceAlerts" } } ``` ```json { "o365audit": { "ClientIP": "67.43.156.13", "CorrelationId": "fe71359f-005f-9000-7cb1-ccf5124703db", "CreationTime": "2020-02-14T18:25:45", "EventData": "