search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
506 stars 397 forks source link

Feature Request: [LLMObs] Support Automatic spans for `langchain_community`'s tools and retriever. #9423

Open kimwonj77 opened 1 month ago

kimwonj77 commented 1 month ago

Summary of problem

We're using Langchain community's tools via langchain.agents.load_tools.load_tools, and it seems like dd-trace-py is not automatically adding spans to it. Just like Langchain's Workflow and LLM spans, automatic integration with Langchain Community tools is needed for tracing. (And also, an option to turn it on and off, if possible.)

Added on May 31: We also discovered that Langchain Community's retriever does not have automatic spans. Additionally, the function langchain.tools.retriever.create_retriever_tool makes it difficult to add spans to the vectorstore/retriever.

Which version of dd-trace-py are you using?

538a0244aca41c032a06519fc84f7aa406f92115

Which version of pip are you using?

Which libraries and their versions are you using?

`pip freeze` ddtrace @ git+https://github.com/datadog/dd-trace-py@538a0244aca41c032a06519fc84f7aa406f92115 langchain==0.1.16 langchain-community==0.0.34 langchain-core==0.1.46 langchain-openai==0.1.4 langchain-text-splitters==0.0.1 langchainhub==0.1.15 langdetect==1.0.9 langsmith==0.1.51 openai==1.23.6

How can we reproduce your problem?

Here's an example from their documentation: https://python.langchain.com/v0.1/docs/integrations/tools/search_tools/

Added on May 31: Here's Retriever too. https://python.langchain.com/v0.1/docs/use_cases/question_answering/conversational_retrieval_agents/

What is the result that you get?

image

What is the result that you expected?

Tools should be on between openai.request

emmettbutler commented 1 month ago

Thank you for reporting this behavior, @kimwonj77. We'll look into it.

cc @Yun-Kim

Yun-Kim commented 1 month ago

Hi @kimwonj77, thanks for making this feature request! The LLM Observability team is looking into this and will likely be able to get this work started in Q3 - please keep an eye out in the upcoming months.

kimwonj77 commented 1 month ago

Thanks for the response!

I also added information about the Retriever (Vectorstore) span, as we discovered that adding a span to the retriever is not straightforward (we had to 'patch' it ourselves). I believe handling this within dd-trace-py would be more appropriate. Therefore, I added this information to the same issue as it appears to be related.

If you need a separate issue for the retriever, I will do that.