Describing schema for Span and Event's payload represented in attributes

[scheler]

Creating this issue to discuss whether it is possible to define the schema for different kinds of Spans and Events/Logs using the semantic conventions as available currently.

What are you trying to achieve?

In the RUM SIG we are trying to define the set of attributes expected for each kind of Span and Event, emitted by the instrumentations for browser and mobile apps. While the semantic conventions define the different possible names under a namespace and their semantics, what is not clear is how the attributes from different namespaces come together to form different types of Spans and Events/Logs. As an example, the span consider a HTTP request, an RPC request and a DB request. They have attributes from at least 2 namespaces - http/rpc/db and net. Although the attributes that commonly go in an Span for HTTP request is listed here, it is not in a machine-readable format such as json-schema, xsd11 or avro. For RUM, this is needed for the purpose of validation on the receiver.

What did you expect to see?

In https://github.com/open-telemetry/opentelemetry-specification/pull/2933, we attempted to list the schema for a few type of RUM events. We are wondering if there is a way to represent the schema in OpenTelemetry specification currently. If not, what is the way to go about introducing the schema concept.

One idea is to borrow the idea of dataschema from CloudEvents specification. We can introduce semantic convention for an attribute called dataschema or otel.dataschema, which points to a URL or URI of a schema for the payload represented by Span/LogRecord Attributes. Further, we can define the specific schemas (versioned and in a standard form such as json-schema or avro) either in this specification repo or some new repo. If this is not of interest to instrumentations outside of RUM, the schemas can reside in a few language repos.

[arminru]

Although the attributes that commonly go in an Span for HTTP request is listed here, it is not in a machine-readable format

The attribute tables in the markdown files you are referring to are actually generated from a machine-readable source. You can find the YAML definition in the https://github.com/open-telemetry/opentelemetry-specification/tree/v1.15.0/semantic_conventions subfolder, e.g., https://github.com/open-telemetry/opentelemetry-specification/blob/v1.15.0/semantic_conventions/trace/http.yaml for HTTP. The tooling for generating these markdown tables as well as code constants can be found in https://github.com/open-telemetry/build-tools/tree/main/semantic-conventions.

Also span events can be described using this model, see https://github.com/open-telemetry/opentelemetry-specification/blob/v1.15.0/semantic_conventions/trace/rpc.yaml#L192 and https://github.com/open-telemetry/opentelemetry-specification/blob/v1.15.0/semantic_conventions/trace/trace-exception.yaml#LL4.

It should be possible to extend the data model and tooling to cover log records and thus RUM events.

For RUM, this is needed for the purpose of validation on the receiver.

Which information would additionally be needed to define and validate RUM events? What kind of validation are you referring to?

[scheler]

@arminru I am aware of the yaml files. By machine-readable, I meant for the purpose of schema validation. and the type of validation we are looking for is contains - we want to validate on the receiver what attributes are expected to be present in each type of Span and Event/Log. Is http.yaml supposed to be used for that purpose too? Typically, schema objects have a name, and we use that to pick the correct schema for a message to validate.

The rpc message events and exception events fit the model expected, so the question in this case is whether this was intentional. How do we go about building tools for contains validation?