open-telemetry / opentelemetry-specification

Specifications for OpenTelemetry
https://opentelemetry.io
Apache License 2.0
3.76k stars 889 forks source link

Define a way to specify more than one Resource in telemetry data #579

Open tigrannajaryan opened 4 years ago

tigrannajaryan commented 4 years ago

Currently it is possible to refer to a Resource in spans and metrics. The Resource indicates where the signal was generated.

However, we may need a way to reference the second (or third, etc) resources.

For example an application A (Resource 1) may report a metric such as "request count" and it may be the number of requests that application A made to application B (Resource 2). Thus, this requires that the metric references multiple resources.

See discussion context here: https://github.com/open-telemetry/oteps/pull/97#discussion_r418747530

Oberon00 commented 4 years ago

See also issue on HTTP app reporting (or per span service resource depending how you look at it): https://github.com/open-telemetry/opentelemetry-specification/issues/335 Also related: https://github.com/open-telemetry/opentelemetry-specification/issues/274 "Allow resources as span attributes"

jmacd commented 4 years ago

Related content here: https://github.com/open-telemetry/oteps/issues/78

To support resources as an API-level concept, as in OTEP 78, requires protocol support for "more than one Resource" as you say.

tigrannajaryan commented 2 years ago

One new use case for this is eBPF. Network monitoring collector needs to record telemetry about 2 entities communicating, each entity can be one of the Resource types that we record using semantic conventions (e.g. network traffic between 2 pods). It is currently not possible to do since there can be only one set of semantic conventions per type of entity recorded in the Resource. Even if we could record 2 sets of conventions in the Resource it doesn't seem to the right approach, since the Resource is supposed to be the indicating the origin of the telemetry.

duxing commented 2 years ago

@tigrannajaryan totally agree with what you said. any application that has network IO to other entities (service, cache, db, etc) should really reflect the traces with 2 entities per communication. one for the client (self, source of network io) and one for the client's observation of server entity (destination of network io).

grepory commented 1 year ago

This has come up a lot, and I continue to be a bit of a purist for Resources that a Resource is something that is the "source" of telemetry.

A metric from ServiceA, RequestCount, is--in an of itself--the number of external requests that ServiceA may make.

Destination is an attribute/label/dimension of the RequestCount metric.

If ServiceB and ServiceC are Resources on the ResourceMetrics, how do you differentiate between multiple RequestCount metrics within a ResourceMetrics blocks as to which count is for ServiceB and which is for ServiceC? Do you duplicate the data?

Even in the case of network flows, you may be observing bidirectional communication, but I think the existing ResourceSpan data model doesn't overly complicate things. You can just as easily have Resource be the observer of the flow and then record source and destination within your trace spans.

We're also more likely to only have partial resources for external entities for attributes that are determined at runtime. E.g. service vs. service instance.

danielgblanco commented 4 months ago

@tigrannajaryan we're assuming you're the sponsor for this, but please do let us know if that's not the case. Thanks.

tigrannajaryan commented 4 months ago

@danielgblanco yes, I am. We are going to address this one way or another in Entities SIG.