search

traceloop / openllmetry-js

Sister project to OpenLLMetry, but in Typescript. Open-source observability for your LLM application, based on OpenTelemetry
https://www.traceloop.com/openllmetry
Apache License 2.0
260 stars 25 forks source link

🚀 Feature: re-write LlamaIndex instrumentation to use LlamaIndex `CallbackManager` #134

Open nirga opened 7 months ago

nirga commented 7 months ago

Which component is this feature for?

LlamaIndex Instrumentation

🔖 Feature description

Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. LlamaIndex has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.

🎤 Why is this feature needed ?

Support LlamaIndex entirely and be future-proof to internal API changes

✌️ How do you aim to achieve this?

Look into LlamaIndex callback_manager and how other frameworks are using it.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

Are you willing to submit PR?

None

kartikay-bagla commented 6 months ago

@quest-bot embark

quest-bot[bot] commented 6 months ago

⚠️ There's no active Quest for this Issue.

Check the docs for more info.

kartikay-bagla commented 6 months ago

@nirga I was looking at the LlamaDebugHandler (in the python code) and it looks like in terms of features, I just need to recreate this and extend it to add the existing traces. Does that sound correct?

nirga commented 6 months ago

Yep @kartikay-bagla. It's a bit trickier cause you need to create and close spans accordingly, and make sure to nest them correctly.

kartikay-bagla commented 6 months ago

Yeah, I'll get started on this because I think it'll take some time.

kartikay-bagla commented 6 months ago

Small update, with this being a larger code change than expected, and some unscheduled life things, I might have to put this on hold until the 18th. Will keep you updated though.

kartikay-bagla commented 6 months ago

So I was able to make some progress over the weekend, but it is slow as I couldn't find any existing implementation of callback_manager that I could use as a starting point.

nirga commented 6 months ago

Thanks so much @kartikay-bagla! I've been in touch with the folks at LlamaIndex so ping me here or on slack if you have any questions I can pass along.

JCaraballo113 commented 5 months ago

Hi, I will try to solve this issue. I do have one question. This feature is desired for the JS SDK or the Python SDK or both?

nirga commented 5 months ago

For both @JCaraballo113 :) @kartikay-bagla are you still on this?