Proposal: Add support for Elastic Common Schema (ECS) in OpenTelemetry

[alolita]

This OTEP is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations.

Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Log Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals. Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTEL Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition.

Please see attached document for the full proposal.

Doc: https://docs.google.com/document/d/1y63W66EyobrnCa9BNZjKzWfETyLMlhC5FiEJzGzaeWU/edit?usp=sharing

Look forward to comments, feedback from the OTEL community. Please join in for initial review of this proposal in the Logs SIG meeting on Feb 23 2022.

Thanks @cyrille-leclerc, Daniel Khan, Jonah Kowall, @kumoroku and others for collaborating on this initial proposal.

[arminru]

Thank you for the proposal, Alolita and others! I read through the document and understand the motivation and advantages but still fail to comprehend what the proposed approach or solution is. Will the outcome of this be a mapping from data (data types, built-in fields, attribute keys) following ECS to the OTel data model and semantic conventions? Will this mapping be implemented in the OTel collector so it can ingest ECS data and then process and export it just as any other data that would follow the OTel model and semconv in first place? Will this mapping be bidirectional so data collected from OTel sources can be exported (by the collector) as ECS data? How are entities present in ECS but missing in OTel treated? Will the missing ones be added to OTel semconv so everything can be mapped or is the intention that those are left untouched? Or is the intention entirely different and you propose to adapt (rewrite) the OTel semantic conventions to entirely follow ECS instead? If so, would this be constrained to Logs only or should this be extended to all signals? I would assume the latter as having separate data models could likely lead to complexity and confusion and that consistency would be desirable instead. We also need to think about the OTel resources here that logs share with other signal types. What would the next action items for implementing this proposal be?

[arminru]

Once you deem the proposal complete it would be great if you could open a PR for your OTEP so it can be reviewed and discussed by the community (but Google Docs is fine while you're still drafting it as it makes editing by multiple collaborators easier for you I assume). Thanks!

[cyrille-leclerc]

Hello @arminru , thanks for your support.

The primary outcomes we are discussing are :

• To enrich OpenTelemetry Semantic Conventions with the modeling defined by the Elastic Common Schema (ECS). Note that we acknowledge the existence of some overlaps/conflicts between the two schemas and we want to resolve those.
• Then we want to have the enriched Otel Semantic Conventions becoming the new schema of the Elastic ecosystem, becoming the kind of next generation schema

Not in scope for the moment:

• Producing converters between ECS as of today and OpenTelemetry Semantic Conventions as of today (e.g. Otel Collector receiver or exporter for ECS). We don't exclude that it could be a goal for a subsequent milestone.

Or is the intention entirely different and you propose to adapt (rewrite) the OTel semantic conventions to entirely follow ECS instead? As explained above, rewriting the Otel Semantic conventions introducing many backward incompatible changes is definitivly not the goal.

How are entities present in ECS but missing in OTel treated? Will the missing ones be added to OTel semconv so everything can be mapped or is the intention that those are left untouched?

As described above the goal is to enrich Otel Semantic Conventions with entities present in ECS and not yet covered by Otel.

... as having separate data models could likely lead to complexity and confusion and that consistency would be desirable instead.

We at Elastic share your point of view, we would like to contribute to the enrichment of Otel Semantic Conventions and then adopt these Enriched Otel Semantic Conventions as the new schema of the Elastic ecosystem.

Did I clarify the goals?

Once you deem the proposal complete it would be great if you could open a PR for your OTEP so it can be reviewed and discussed by the community (but Google Docs is fine while you're still drafting it as it makes editing by multiple collaborators easier for you I assume). Thanks!

Moving to a collaboration using GitHub Pull Requests is the plan. We have to define a process, we discussed the idea of an incremental approach to integrate ECS namespaces one after the other. The methodology has to be clarified.

[cyrille-leclerc]

@arminru we have created the PR, please feel free to comment and contribute:

• https://github.com/open-telemetry/oteps/pull/199

[AlexanderWert]

We created a new PR to address this proposal: https://github.com/open-telemetry/oteps/pull/222

[trisch-me]

Should we close this issue? The final otep was merged long time ago and donation of ECS into Otel is in progress