search

traceloop / openllmetry

Open-source observability for your LLM application, based on OpenTelemetry
https://www.traceloop.com/openllmetry
Apache License 2.0
1.52k stars 113 forks source link

🚀 Feature: re-write LlamaIndex instrumentation to use LlamaIndex `CallbackManager` #540

Open nirga opened 4 months ago

nirga commented 4 months ago

Which component is this feature for?

LlamaIndex Instrumentation

🔖 Feature description

Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. LlamaIndex has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.

🎤 Why is this feature needed ?

Support LlamaIndex entirely and be future-proof to internal API changes

✌️ How do you aim to achieve this?

Look into LlamaIndex callback_manager and how other frameworks are using it.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

Are you willing to submit PR?

None

varaarul commented 3 months ago

I can start looking into this!

larinam commented 2 months ago

@nirga , could you fix the link to documentation in the description and give a hint what pieces of this project's code should be touched by this change?

nirga commented 2 months ago

Done @larinam! Basically, right now we wrap LlamaIndex calls to log and create spans. This means we have to keep up with their APIs and update our instrumentations every time something changes on LlamaIndex's end.

A better solution would be to register ourselves to the callbacks of the LlamaIndex engine and just open and close spans when we get called. This is much more robust and future proof.

larinam commented 1 month ago

All right @nirga , thanks. But I'll try to ask one more time with the question. The code base is not that little for the project, could you give a hint what pieces of this project's code should be touched by this change? Could you just attach a couple of links?

nirga commented 1 month ago

In our project it's just under the LlamaIndex instrumentation. Do you want to join our slack and discuss there? Might be easier :)