Add KeyValue support for Observation.Event

[lenin-jaganathan]

Please describe the feature request. Currently, Observation.Event supports only providing a name (and contextual name). The enhancement request is to allow the addition of low cardinality and high cardinality tags (aka attributes) to events.

Rationale This would allow the Observation handlers to use them appropriately for use cases like tagging the counters created from the events with more informative tags, key value-based spans/logging systems can use this to enhance search capabilities and enrich events with additional contextual information, etc. Adding support for tags in events would also enable places where certain tags are associated with an event to be flushed immediately and remove references. Today, such information can only be added to Observation which might be long-lived. Also, we cannot add too many low cardinal tags to the observation as that would make it emit timers with too many attributes. Doing so in events would ease these off from timers and add to counters which are relatively cheaper compared to Timers.

[jonatan-ivanov]

Thanks for the issue! Could you please provide us a concrete real-world example where you would use this?

I think providing a simple implementation for this wouldn't be too hard, I'm a bit concerned though about two things:

• Performance: in order to have KeyValues on events, we need to maintain two containers on each of them, I'm not sure how bad is it though, hopefully it's not too bad.
• Breaking implementations of the Event interface: if we add new (non-default) methods to it (getter/setter), existing implementations can be broken, for example if somebody did something like this: GrpcServerEvents

/cc @ThomasVitale @jzheaux

[lenin-jaganathan]

Sure. There are multiple places where we can use the KeyValue associated with an event. I am sharing some of the use-cases with each Observability signals that can be derived from them.

1. Metrics: Currently, a counter is created for each event. This counter includes all the low cardinality tags in the context which seem to add way too many tags to the counter created. For instance, a counter for HTTP "receive" might not need all the information that was added when sending the request. Also, both send and receive might not need all the information associated with HTTP observation. Also, to get additional tags in Counter we need to add the tags in observation which then gets added to all the other events and more importantly to the timer as well, which makes the timer costlier and heavier.
2. Logs: This is where we see more scope for events. We have a transactional logging system where events are tied to transactions. The above case with metrics apply here as well. Especially with high cardinality tags and 10's of events it make sense to add them to events instead of the entire observation itself. Today, adding them to observation makes it very very heavy and it also gets attached to other events if we try to use context to get data. Instead of that, if events support KeyValue we can keep them away from context and make context very lean. Only context gets propagated across thread boundaries and events can be flushed out as when they are created and the data can stay with them instead of being in the entire context. If we try to avoid getting KeyValue from context, events really make little sense with just the name part of them.