search

alexrudall / ruby-openai

OpenAI API + Ruby! 🤖❤️ NEW: Assistant Vector Stores
MIT License
2.6k stars 302 forks source link

OpenTelemetry integration #292

Open jamiemccarthy opened 11 months ago

jamiemccarthy commented 11 months ago

I'm writing an app with ruby-openai and it gives me a bit of a worry that I don't know how much I'm spending on OpenAI. I would like a pane on my Honeycomb.ai dashboard measuring how much data I'm sending and receiving. Adding alerting to my dashboard for unexpectedly high usage would give me peace of mind. And if latency of OpenAI's servers was also instrumented, so I could see that latency in traces, that would help me understand what performance I should expect as well.

OpenTelemetry seems like the natural way to instrument this. Here's a list of existing libraries. I'd love to add

  c.use "OpenTelemetry::Instrumentation::Openai"

to my config/initializers/opentelemetry.rb and have my ruby-openai usage magically become available on my Honeycomb dashboard. Or of course it would work for whatever other provider people are using.

My suggested implementation would be to make the telemetry class a separate gem, and add a check whether the gem's class is defined in OpenAI::HTTP#json_post and related methods. If so, wrap the methods' functionality in a tracing block. But I'm open to alternatives. I haven't actually written a telemetry module before, so maybe thinking it's easy is hubris on my part.

Alternatively, I could add a similar wrapper in my own app and just use it for personal use. But this seems like something that would benefit the community and might as well be shared.

marckohlbrugge commented 11 months ago

One approach is to make the gem more extendable. The HTTP gem does a great job at this. Here’s some examples:

https://github.com/httprb/http/tree/main/lib/http/features

If this gem would have the right hooks, it allow anyone anyone to more easily build features like telemetry on top of it.

alexrudall commented 7 months ago

@jamiemccarthy - v6.3 allows you to add custom Faraday middleware - see README - I think this could be used to link to OpenTelemetry?

alexrudall commented 2 weeks ago

Info on this - https://open-telemetry.github.io/opentelemetry-ruby/opentelemetry-instrumentation-faraday/v0.16.0/ - could be nice to add instructions to the README if people would find it useful

jamiemccarthy commented 2 weeks ago

Cool, I'll see about giving this a try in my project shortly, thanks for the suggestion