search

open-telemetry / semantic-conventions

Defines standards for generating consistent, accessible telemetry across a variety of domains
Apache License 2.0
244 stars 158 forks source link

Remove any possibility to record GenAI prompts and completions as attributes once Event API is available #1278

Closed lmolkova closed 1 month ago

lmolkova commented 1 month ago

See https://github.com/open-telemetry/semantic-conventions/pull/980#pullrequestreview-2164569253

How to check that Event API is implemented:

codefromthecrypt commented 1 month ago

For me this was tough to grok as I had no idea where to look on a per-language basis, except searching for the word event which well, isn't great.

I asked an LLM what the most popular languages for GenAI were and here are the status of the ones it mentioned (besides R which I'm not sure we have a repo for)

Since I personally work in Go and would like to help the uber popular ollama project, I also looked at Go

I don't have karma to edit description, but I would suggest making a checklist so that it is easier for folks to look at this and get a quick gauge.

codefromthecrypt commented 1 month ago

also, while it may be too long an issue title, in the description, it helps to clarify the spec is currently about "span event attributes", not the more troubling "span attributes". Basically, linking to my issue comment helps, but maybe after a description of the now state.

Example description summary, feel free to use, reword or leave as a comment:

The current LLM/GenAI semantics indicate prompt and completion data recorded as (Log) Events, not Span Events.

Right now, python is the most dominant programming language in GenAI, but it doesn't yet have support for the Event API. Even languages that do support it, such as java and javascript, caveat it as incubator or experimental. This means instrumentors may not be able to use the (Log) Event API reliably, especially across languages.

For this reason, the LLM/GenAI semantics provide for a fallback to using Span Events to record the same prompt/completion data as what should use the Event API.

This issue tracks status of the major language SDKs. Once a quorum is met, we should remove the fallback advice so that the Event API, and not Span API is used for prompt/completion data.

jack-berg commented 1 month ago

java "Event API" - available in incubator @jack-berg do you know the criteria or any other context on this graduating from the incubator?

There are two criteria:

  1. The event API spec needs to be stable. One of the required for stabilizing a spec is that there are implementations in at least 3 languages. We like the 3 languages to have sufficiently different programming paradigms to help discover issues / clarifications. There's a dedicated events SIG working on specific problems related to this (see "Specification: Events" here). The group has largely been driven by client instrumentation requirements (browser apps, mobile apps, etc) so having this additional LLM use case would certainly be good to ensure the design isn't overindexed on one domain.
  2. The opentelemetry-java maintainers / approvers have to vet the specific API design / implementation and feel confident promoting it to the stable API package, which comes with significant backwards compatibility guarantees. I wrote a lot of the events API stuff in opentelemetry-java and feel pretty good about it, but we have a problem where the java SIG isn't sufficiently staffed and its hard to get the code reviews we need to make progress. Things don't move as fast as I'd like.

Just to add one more piece of context to this conversation since it may not be known / obvious to all readers: the Event API is likely desirable because its built on top of the log signal, which supports an AnyValue body and attribute values. AnyValue allows the modeling of complex types (i.e. maps of maps, heterogeneous arrays, etc) which are not supported in the standard attribute definition used everywhere else attributes appear in the spec (i.e. resource attributes, scope attributes, metric point attribute, span attributes, span event attributes, span link attributes).

codefromthecrypt commented 1 month ago

Thanks for the comprehensive reply @jack-berg!

... we have a problem where the java SIG isn't sufficiently staffed and its hard to get the code reviews we need to make progress. Things don't move as fast as I'd like.

If you feel alone on a PR you feel important and a review by a not-yet-approver would be helpful, feel free to tag me. Sometimes, a beginning review can help momentum, though not a sure thing.

lmolkova commented 1 month ago

The best way to know if specific API/feature is available (and where) is spec-compliance matrix - https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#events

There is a link to it in the #980

codefromthecrypt commented 1 month ago

There is a link to it in the https://github.com/open-telemetry/semantic-conventions/pull/980

Thanks, maybe include this in a derivation of this issue desc, unless the norms of this repo is to not have PR descriptions if someone can derive the same from a linked comment. Again, I would do this myself instead of nagging you, but I don't have access.

lmolkova commented 1 month ago

I'm closing this issue since #980 does not use span events anymore

codefromthecrypt commented 3 weeks ago

thanks agreed!