search

open-telemetry / semantic-conventions

Defines standards for generating consistent, accessible telemetry across a variety of domains
Apache License 2.0
250 stars 163 forks source link

Messaging: How to describe original destination messages were published from #123

Closed joaopgrassi closed 1 year ago

joaopgrassi commented 1 year ago

What

In open-telemetry/opentelemetry-specification#3450 and its PR #100 we removed the attributes under the namespace source.* because of the highlighted issues with it (mostly because source is a concept/namespace already defined in ECS and we didn't want to clash with it).

During the PR reviews in #100 though, we went "back-and-forth" on the topic, discussing if we even needed to have such conventions or if we should just remove them, since we weren't really sure if they really were a practical use-case.

Context

@dpauls did some research on the topic which I will paste the contents here:

The sort of use case where I think the message's original destination (published destination, producer destination?) is useful is when the broker maps a message's destination onto a consumer destination.

From an client application's point of view, this is easily illustrated with JMS:

Note in particular the documentation for Message.getDestination() says specifically:

When a message is received, its JMSDestination value must be equivalent to the value assigned when it was sent.

This is specifically highlighting that it may not match the MessageConsumer's destination.

JMS Providers where it would be common for these destinations to differ would be providers like Solace PubSub+, IBM MQ, Tibco EMS and AMQP. In the case of JMS over AMQP, they've documented the details of how to carry the JMSDestination in the message's x-opt-jms-dest message annotation. They state that if it isn't present (and the to field of properties is also not present) only then does a message's JMSDestination revert to match the Destination used to create the MessageConsumer.

Rabbit MQ also has this same concept where a message is sent to an (exchange, routing key), which directs the message to one or more queues for consumption. When a RabbitMQ Consumer receives a message in handleDelivery, it receives an Envelope, from which the message's exchange and routing key can be obtained.

I'll also point out that the originally published destination often contains particularly useful information about the message itself. In a loosely coupled Pub/Sub messaging system, the destination the consumer is created with would often be something rather static, perhaps the name of an application using the queue. It's the destinations the messages were sent to that really tell you about the message itself.

Given that standards such as AMQP and JMS have separated these two concepts, and I've listed several messaging systems that really do this, I think it would be beneficial to include this information in the conventions.

I liked the original terminology of "source" and "destination" the best. If this does not align with ECS, and such alignment is important for this group, then destination_original is "good enough" from my point of view. Other possibilities I can think of are hardly much of an improvement (destination_publish[ed], destination_sent). I'd be open to any other naming suggestions but the worst of all worlds is each messaging system that has the concept uses a different name for the same concept.

I don't see much value in removing it. It is optional to begin with and would only be included for messages where it differs from the consumer's destination, so there's no implementation burden for messaging systems where it wouldn't apply.

What we need to do

Given @dpauls research, it seems there's a real use-case for recording such attributes.

The question we need to discuss and answer is: how do we want to name this?

Suggestions that came up during the PR:

dpauls commented 1 year ago

I like destination_publish. It aligns with other terminology (i.e. the operation name when a message is published is publish). By keeping destination at the beginning, it perhaps makes it more obvious that it is related to destination.

joaopgrassi commented 1 year ago

We discussed this in the messaging SIG meeting on 2022/06/29 (meeting notes) - We agreed on destination_publish for the namespace and to bring back the name and anonymous fields under it. I will submitting a PR soon.