[Filebeat] [Cisco module] - Improve ECS utilization

[defendable-forfot]

We are ingesting Cisco Umbrella data into our Elasticsearch for search, detection in Elastic Security and visualization through Kibana. However, we have noticed a few specific fields where the Cisco module does not optimally utilize ECS.

Note: we are running filebeat version 8.1.3, but have noticed that none of the newer releases solves our issues.

Umbrella

cisco.umbrella.identities
    ECS fields: related.hosts | host.name
    Suggestion: Identities, which represent a host, are added to the related.user field. This field should not be used and instead the data should populate host.name and related.hosts. Additionally, the data in source.user.name is also a host and should be removed.
message
    ECS fields: related.hosts
    Suggestion: The message field is currently just a collection of various data concatenated together and utilizes and does not give much value in Kibana. Some of the data in the message field is already present in other data fields as well. One useful improvement here would be to ensure that all hosts, which appear in the message field, are added to the related.hosts field.
destination.domain | related.hosts
    ECS fields: url.domain | url.extension | url.subdomain
    Suggestion: The related URL is currently added to the destination.domain field. This data should be parsed and used to populate the url.domain, url.extension and url.subdomain fields. Domains/URLs should not be added to the related.hosts field.

Additionally, we believe the ECS specification should be improved with the introduction of a new field within the Related fields section. Certain third-party data sources, the Cisco module included, send events where multiple URLs are present. An optimal solution would be to add this data to a related.url field.

This is a copy of https://discuss.elastic.co/t/cisco-filebeat-module-umbrella-improve-ecs-utilization/315127, as I was recommended to post this as a GitHub issue instead.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

The identities issue has come up elsewhere. This is not a trivial issue to solve and we think depends on changes to kibana that would allow user-provided pipelines to allow the user to correctly copy values across to the appropriate ECS field. The problem here is that the Cisco notion of an identity spans multiple concepts in the ECS and is user-configurable, so no single generic solution will fit all cases.

For the message suggestion, can you provide some examples of messages that should be parsed into related.hosts?

For clarity, can you provide an example of the behaviour you are describing for the destination.domain|related.hosts issue?

[defendable-forfot]

At least on our end we have only observed the identities containing actual hosts, but I see how the definition of the field provided by Cisco could lead to further issues as it is not set on being a host specifically.

Regarding the message question: The message field will typically look something like this in our logs: \"2022-00-00 00:00:00\",\"DESKTOP-XXXXXX1\",\"DESKTOP-XXXXXX1\",\"127.0.0.1\",\"8.8.8.8\",\"Blocked\",\"1 (A)\",\"NOERROR\",\"website.url.com.\",\"Malware\",\"Roaming Computers\",\"Roaming Computers\",\"Malware\" Included within the message field is the host related to the observation DESKTOP-XXXXXX1, we believe this could be added to related.hosts. However, the host observed in message, second and third value, is identical to the value in cisco.umbrella.identities. I'm unsure how message may look like if other values were to be provided by Cisco within the identities field.

Regarding the destination.domain|related.hosts question: In the same document that is used as the example above, we can see that the value in the destination.domain: website.url.com also populates related.hosts for some reason. To specify: the value found is related.hosts is website.url.com.

[efd6]

Would you be able to provide a sanitised event for the example above rather than a hand-crafted one; I am unable to parse the construction that you have there.

[defendable-forfot]

Sorry for the somewhat late response, hopefully this is better:

"2022-10-12 10:19:07","Company Hostname","Company Hostname","192.168.0.0","10.10.10.10","Blocked","1 (A)","NOERROR","malicious.com.","Advertisements,Malware","Roaming Computers","Roaming Computers","Malware"

[efd6]

Can I confirm that the background of this is that you are running filebeat directly and not via agent? I am unable to get the cisco_umbrella integration to ingest this message, but can get the filebeat cisco module to ingest using the umbrella fileset if I let it know that the message is a dnslogs message. In that case I get this document

{
    "@timestamp": "2022-10-12T10:19:07.000Z",
    "cisco.umbrella.blocked_categories": "Malware",
    "cisco.umbrella.categories": "Advertisements,Malware",
    "cisco.umbrella.identities": [
        "Company Hostname"
    ],
    "cisco.umbrella.identity_types": "Roaming Computers",
    "cisco.umbrella.policy_identity_type": "Roaming Computers",
    "destination.domain": "malicious.com.",
    "dns.question.type": "1 (A)",
    "dns.response_code": "NOERROR",
    "dns.type": "query",
    "event.action": "dns-request-Blocked",
    "event.category": "network",
    "event.dataset": "cisco.umbrella",
    "event.module": "cisco",
    "event.original": "\\\"2022-10-12 10:19:07\\\",\\\"Company Hostname\\\",\\\"Company Hostname\\\",\\\"192.168.0.0\\\",\\\"10.10.10.10\\\",\\\"Blocked\\\",\\\"1 (A)\\\",\\\"NOERROR\\\",\\\"malicious.com.\\\",\\\"Advertisements,Malware\\\",\\\"Roaming Computers\\\",\\\"Roaming Computers\\\",\\\"Malware\\\"",
    "event.type": [
        "connection",
        "denied"
    ],
    "fileset.name": "umbrella",
    "input.type": "log",
    "log.offset": 0,
    "message": "\"2022-10-12 10:19:07\",\"Company Hostname\",\"Company Hostname\",\"192.168.0.0\",\"10.10.10.10\",\"Blocked\",\"1 (A)\",\"NOERROR\",\"malicious.com.\",\"Advertisements,Malware\",\"Roaming Computers\",\"Roaming Computers\",\"Malware\"",
    "observer.product": "Umbrella",
    "observer.type": "dns",
    "observer.vendor": "Cisco",
    "related.hosts": [
        "malicious.com."
    ],
    "related.ip": [
        "10.10.10.10",
        "192.168.0.0"
    ],
    "related.user": [
        "Company Hostname"
    ],
    "service.type": "cisco",
    "source.address": "192.168.0.0",
    "source.ip": "192.168.0.0",
    "source.nat.ip": "10.10.10.10",
    "source.user.name": "Company Hostname"
}

Is your suggestion that we should be putting the contents of related.ip into related.hosts as well?

[defendable-forfot]

Yes we are running Filebeat directly and not through Agent.

In regards to the second question, no, I am not suggesting that the contents of related.ip should populate related.hosts. "Company Hostname" should populate related.hosts instead of the domain, as "Company Hostname" is an actual host.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!