Please explain difference between Core and Extended

[MikePaquette]

What is the difference between “core” and “extended” levels? (A definition for each type would be helpful here.) Is “core” required? Do you generate an error when core fields are missing?

No. 2 of 16. This question was asked by a new ECS user, who is familiar with mapping IT events to data models and use cases in other schemas. These questions are being posted as a GitHub issue, because a) they may offer valuable insights. b) we expect that many new users will have similar questions.

[MikePaquette]

NOT OFFICIAL ECS DEFS:

ECS-Core Fields: A common set of generalized fields that SHOULD be populated where applicable for all data sources in an ECS-compliant implementation. ECS-core fields are used in common operations such as aggregations, visualizations, filtering, and analysis. For example dashboards built with these fields should be able to be re-used on virtually any ECS-compliant implementation.

ECS-Extended Fields: A general name given to additional ECS fields that are not the ECS-Core fields. ECS-Extended field names are specified with an implied hierarchy based on the top level namespaces/objects/prefixes that prefix them. All ECS-Extended fields exist under the same ECS Top-level namespace/objects defined in the README.md https://github.com/elastic/ecs/blob/master/README.md#fields.

[webmat]

Here are other ways we've defined "Extended" fields, depending on the situation:

• New fields that may one day move to Core, but whose semantics are not as fully fleshed out as Core. This implies that the risk of breaking changes in Extended fields may be slightly higher than Core.
• Fields that may be very important to a given use case, but is not applicable or considered important outside of that specific use case.

I agree the definitions of / distinction between Core and Extended needs some attention :-)

[ebeahan]

Now covered in the ECS docs: https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html#_ecs_field_levels

[richardgilm]

Could we have a list of such "Core" Fields? It's not clear or evident what these fields are. For instance, logs.level or event.dataset or kubernetes.pod.uid or aws.s3.bucket.name . Which ones are core?

[djptek]

Hi @richardgilm the distinction between Core and Extended is elaborated in the ECS Guidelines

To see an exhaustive list of all current Core and Extended fields, see Column 6 of the full list here Generated ECS Fields

so

log.level
event.dataset

are Core fields

At this point in time neither

kubernetes.pod.uid
aws.s3.bucket.name

are ECS fields, however, for k8s you might want to consider the Orchestrator and Container field set to capture these

[richardgilm]

Yet these fields that are not Core do appear as necessary for the Logs UI or Metrics UI https://www.elastic.co/guide/en/observability/current/logs-app-fields.html https://www.elastic.co/guide/en/observability/current/metrics-app-fields.html

Hence the confusion

[djptek]

@richardgilm the Logs App fields page notes: "Please note that some of the fields listed are not ECS fields"