lmolkova
commented
4 months ago Adding @SergeyKanzhelev who might be interested in GCP server LLM metrics.
The assumption is that client and server would have different information available.
E.g. gen_ai.client.operation.duration is different from gen_ai.server.operation.duration - depending on the client network, the difference can be significant.
The same request that's timed out on the client could be successful on the server side resulting in different duration, error rate, usage and other metrics.
Client and server metrics might have different attributes. E.g. server might have information about pricing tier, region, availability zone that is not available on the client, but are very useful to know.
Therefore we either need:
- different metrics names.
- extra attribute(s) that distinguish client from server.
Other otel semantic conventions use option 1 and this is a good reason for gen_ai semconv to also defined different metrics for client and server.
But nothing stops us from adding gen_ai server semantic conventions and reusing anything we can between client and server.
gyliu513
Area(s)
area:gen-ai
What happened?
Description
There is a PR trying to enable vllm support metrics as well and it is adopting this semantic convention as well https://github.com/vllm-project/vllm/pull/4687 , here the question is vllm is server side, but I can see seems now the llm semantic convention are manily trying to instrument client side code like https://github.com/traceloop/openllmetry/tree/main/packages, do we need to distinguish the client side and server side semantic convetion? Thanks!
@nirga @lmolkova ^^
Semantic convention version
NA
Additional context
No response