LLM semconv: how to capture prompts and completions

[lmolkova]

Area(s)

llm, events

Is your change request related to a problem? Please describe.

Prompts and completions are essential part of debugging experience when developing LLM applications. They could be verbose and contain sensitive information, so they need some opt-in mechanism such as feature-flag, configuration property on the instr library. Specific OTel distros, debugging and diagnostic tools within IDEs may decide to enable the collection by default.

There are multiple ways to capture LLM events: as attributes on spans, as attributes on span events, as log events.

1. Span Attributes:

   • span attributes are limited in size (not on the OTel SDK, but on the vendor side) and could be as small as few kilobytes
   • they are not structured and we'd have to capture the format (e.g. mime type) to allow parsing them
   • but they always stay along with the span making visualization tooling and queries easier. Users can't forward them to a separate backend along with the logs.

2. Span event attributes:

   • span event's don't have a notion of body and are not structured either
   • event attributes are subject to the same problems as span attributes.
   • they are also exported along with the span and arguably are easy for visualization tooling to use
   • their future is uncertain - see https://github.com/open-telemetry/semantic-conventions/issues/695 and https://github.com/open-telemetry/opentelemetry-specification/issues/3406

3. Log events/logs:

   • Log events have a notion of body - tracing/logging vendors usually have bigger limits for it than for attributes
   • Log events are structured. We can specify the structure for each event and simplify parsing/querying
   • They are exported separately from spans, but are still correlated with them. If user sends logs and traces to different backends, LLM-debugging experience would need cross-vendor queries which would be challenging.
   • They are not implemented/released/stabilized by most of the languages, eventlogger implementations can only be found in PHP and C++

Since logs/event logs are not ready to be used by instrumentation libraries in popular languages used by LLM applications (python, JS), at least the initial version of LLM semconv cannot use them leaving us with attributes vs span events choice.

Describe the solution you'd like

LLM semantic conventions should allow to capture prompts and completions as span events, but also be future-proof and support new log events:

• I'd like to be able to describe LLM event structure/format in the semantic conventions
• When they are reported as span events, I want to have a designated attribute to capture a body such as event.body (similarly to event.name).
• Same with severity number (see #828)

Once/if span events are deprecated, the same conventions could be used to provide back-compat story for the instrumentation libraries or OTel SDKs themselves.

Vendors/exporters may decide to transform such events into log records and would map event.body to the corresponding property with higher length limits than attributes

While these are captured as span events and are sent over the wire along with the spans, vendors/exporters may decide to extract prompts/completions from the event attributes and put them on spans directly to simplify their visualizations and queries.

[lmolkova]

@MSNev could you share your take on this?

TL;DR: since we cannot immediately use event API in the instrumentation libraries, I want to add event.body attribute and maybe severity number that would bridge the gap between span events and log events. Then I'd like to be able to describe LLM events as good-new-structured-events in semconv with fallback to event.body for the time being until clarity on span events is reached.

[MSNev]

I want to add event.body attribute

We actually talked about this a few weeks back as a possible path forward for "span event" support, but as spans only support "standard attributes" (non-nested) any value within this attribute will need to be encoded.

So yes I would support the introduction of an event.body attribute (in the global semantic registry) for use within a span event. We will need to determine what the encoding within this attribute would be -- I would (default) to a JSON encoding, but then should these be OTLP attribute objects or just "normal" objects as the Log body encoding of the "body" being represented as OTLP is really a side effect of the OTLP transport.

tagging @tedsuo and @trask for visibility and any additional thoughts.

[lmolkova]

Thanks for the context!

The event.body would have to be a string and there is no way to enforce JSON there.

Maybe this might work:

• Specific event structure is documented in semconv.
• There is 1:1 relationship between event.name and that specific structure
• Semconvs decide which format should be used in the event.body.
  • One may use plain text, another can say it's a json
• Instrumentation lib writes event with that structure to event.body
• If a backend/user wants to convert this event into a structured log body, it has to be aware of the structure documented in the semconv for this event.name. Otherwise it can default to a plain text.

This requires backends to be aware of specific events and their structure if they want to provide any experience on top of them, but it seems it would self-heal with structured log events. We also keep the door open to add event.body.content_type (which I hope we won't need, but the option still exists, we might consider using event.body.text though to keep the body as a namespace)

[lmolkova]

Ah, I see the problem that not just the backend, but span event -> log event converter inside OTel SDK would have to know the structure. And if we default to json this problem becomes easier since we can deserialize to json tree and put the object onto otlp.

I think we can try starting with json as a requirement and if we find we need to support plain text, we'd add mime type attribute? (or the span_event -> log_event would check if it's json and fallback to plain text if it can't deserialize - terrible for perf in general case though)

[lmolkova]

Fixed in #980