search

jackmpcollins / magentic

Seamlessly integrate LLMs as Python functions
https://magentic.dev/
MIT License
1.79k stars 81 forks source link

Support OpenTelemetry #136

Open jackmpcollins opened 3 months ago

jackmpcollins commented 3 months ago

Each function created using a magentic decorator (@prompt, @chatprompt, etc.) should have a corresponding span, as if it was also decorated with @tracer.start_as_current_span(). The function arguments (with strings truncated) should be set as attributes on the span. The function name and a hash of the template string/messages should also be set as attributes on the span so that filtering by a function or a specific prompt template is possible.

The template hash should be available to users as a .template_hash or similar method/property on their magentic functions so this is easily accessed. This should use hashlib.sha256 to be stable across Python versions and reproducible outside of magentic.

Documentation for the opentelemetry integration should include instructions on how users should wrap the functions they pass to the functions argument. This is most useful for @prompt_chain because it will make which function(s) the model decided to call visible in tracing. Maybe magentic should do this automatically?

Documentation for this should also include showing the traces in a UI, ideally one that can be easily installed and run locally. Looks like https://github.com/jaegertracing/jaeger is ideal.

Users should be able to configure whether magentic functions create spans, and whether/which function arguments are added as attributes to the spans.

patcher9 commented 1 month ago

Hey @jackmpcollins Coming here from your reply on our Hacker News Post

I think there are two ways you can go about this!

  1. I can try to see if this is something we can add in our OpenLIT SDK, This way the observability part can stay external. THough no promises on feasability yet as I haven't checked the full code in this repo.

  2. I can assist you in making sure the span attributes are compatible with OpenTelemetry Semcov for GenAI and what users can use to monitor and track in the OpenLIT UI if needed. (They can choose to view it in any of the OTLP compatible tools also)

Lemme know whatever is your preferred approach and I can then start evaluate of the best way to go with this, Thoughts?