search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.76k stars 667 forks source link

[Feature Request] OpenLLMetry Integration (OpenTelemetry) #1039

Open amitjoy opened 3 months ago

amitjoy commented 3 months ago

Describe the Feature OpenTelemetry is a de facto standard for reporting metrices and OpenLLMetry uses OpenTelemetry under the hood. It would be better to have an integration to send such metrices for RAG evaluation to OpenTelemetry collectors as well.

Why is the feature important for you? Since OpenTelemetry is a common standard which would be used easily in any existing systems.

Additional context NA

jjmachan commented 3 months ago

hey @amitjoy thanks for the feat request

I have been keeping an eye on OpenLLMetry for a while and with

there should be some progress and it should be easier. I'll take a closer look at this and see what would the easiest 🙂

in the mean time do you use OpenLLMetry for your LLM application tracing also? do you use any other tools to visualize the traces?

amitjoy commented 3 months ago

Hi @jjmachan,

Thank you for your prompt response and the detailed information regarding the integration with ragas. Yes, I do use OpenLLMetry and publish the metrics and spans to Sentry, as it can also function as an OpenTelemetry Collector. Since I already utilize Sentry for monitoring unexpected runtime issues, it seemed logical to extend its use to OpenTelemetry traces as well.