Provide Canonical Mappings in ECS Documentation

[andrewthad]

Summary

Canonical mappings for well-understood devices or systems should be part of the ECS documentation. This means devices with a stable and documented log format. For example: zeek, suricata, haproxy, palo alto firewalls, fortigate firewalls, iptables, standard SNMP mibs, windows event logs. This will make it easier for people to figure out how to apply ECS to their workload. Even a user of a device without a canonical mapping would benefit from looking at how a similar event (or an event from similar device) is mapped.

Motivation:

To my knowledge, there are several ways to figure out how to map logs and events to ECS:

• Search through the issue tracker to see if someone has provided an example mapping for the logs from the particular device you are working with.
• Look for the mapping that an elasticbeat uses.
• Look through all the fields until figuring out which one describes your event.

ECS appears to been originally authored with firewalls front and center. There are even some examples of how to map these on the issue tracker, but some of them are slightly wrong or out of date. Events that aren't firewall events can be hard to figure out because there is less discussion about it.

Rather than being relegated to the issue tracker, this information should be part of the ECS documentation. That way, as new fields are added, old examples can be updated.

Detailed Design:

Only devices and software with documentation that provides names for fields should be considered. Without a vendor-supplied canonical name, it is difficult to refer to a field. Every mapping should include:

• At least two example logs (anonymized)
• A hyperlink to the device-specific documentation that describes the log format (not a link to an ECS page)
• A table with the columns: original field name, ecs field name (hyperlinks to ECS page documenting field), notes.

Implementation Plan

I have already developed several mappings that I am able to contribute if this proposal is accepted.

[ypid-geberit]

From what I can see, this already exists but in another place:

• https://github.com/elastic/beats/tree/master/x-pack/filebeat/module/
• https://github.com/elastic/beats/tree/master/x-pack/filebeat/module/suricata

[andrewthad]

I don't see mappings on either of the hyperlinked pages. Is there a more specific resource in that repository?

[ypid-geberit]

https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/suricata/eve/ingest/pipeline.yml

[ebeahan]

Not exactly the full mappings proposed in the original description, but there's an effort to include more real-world usage examples to the ECS documentation. Here are a couple:

• https://www.elastic.co/guide/en/ecs/current/ecs-user-usage.html
• https://www.elastic.co/guide/en/ecs/current/ecs-mapping-network-events.html

The intent is to have both:

• Sections that act as a reference which includes the field's description, type, example values, etc.
• Sections that describe how to populate particular fields, relationships with other fields, and include mapping examples from actual log sources. This will not likely map an entire event, but the aim is to better demonstrate foundational concepts for users to apply to mapping their own sources.