search

traceloop / openllmetry

Open-source observability for your LLM application, based on OpenTelemetry
https://www.traceloop.com/openllmetry
Apache License 2.0
1.96k stars 199 forks source link

🐛 Bug Report: Watsonx traceloop flow has many http request spans which is distracting #542

Closed huang-cn closed 7 months ago

huang-cn commented 7 months ago

Which component is this bug for?

Watsonx Instrumentation

📜 Description

when using traceloop flow with watsonx, the trace data include many http request spans, which is distracting.

👟 Reproduction steps

following the traceloop openllmetry example to instrument watsonx LLM, and output otel data to jaeger.

👍 Expected behavior

plan for improvement:

  1. adding these domain names to traceloop-sdk tracing EXCLUDED_URLS to avoid generating http request spans
  2. create a watson model init span to show the model initialization response time.

👎 Actual Behavior with Screenshots

image

🤖 Python Version

3.11

📃 Provide any additional context for the Bug.

No response

👀 Have you spent some time to check if this bug has been raised before?

Are you willing to submit PR?

Yes I am willing to submit a PR!

paolorechia commented 7 months ago

My first impression is that this is not a bug but intended behavior. That being said, making it more configurable is always a good idea

huang-cn commented 7 months ago

@paolorechia It appears there's no simple switch to turn these http request spans off, in the sdk tracing EXCLUDED_URLS we can see URLs of many api have been excluded already, we're planning to add the Watsonx api to the EXCLUDED_URLS to address this.

paolorechia commented 7 months ago

I see, I found it (https://github.com/traceloop/openllmetry/blob/3eba03e21b99cfd9b09bc46cdd1d009634f43072/packages/traceloop-sdk/traceloop/sdk/tracing/tracing.py#L35)

Makes sense to me, I agree you can consider it a bug then 🙂

paolorechia commented 7 months ago

@nirga While researching for semantic conventions I’ve seen some OpenTelemetry instrumentations define trace level.

Maybe it’s a feature to consider for traceloop, so users can adjust these kind of things?

paolorechia commented 7 months ago

Actually it’s something called span.kind - not sure if applicable

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/8daa8ad48108775d8e799a3abc3ed06f84b4c00e/instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/instrumentation/flask/__init__.py#L349

nirga commented 7 months ago

@huang-cn yep, sounds good! @paolorechia that's a good question, which span kind to use. We're starting a semantic convention working group to discuss that - https://github.com/open-telemetry/community/blob/main/projects/llm-semconv.md