newrelic / node-newrelic

New Relic Node.js agent code base. Developers are welcome to create pull requests here, please see our contributing guidelines. For New Relic technical support, please go to http://support.newrelic.com.
Apache License 2.0
963 stars 391 forks source link

(langchain): Instrument chains #1965

Closed jsumners-nr closed 5 months ago

jsumners-nr commented 6 months ago

There are at least two chain concepts in LangChain:

  1. Direct chains as described in https://js.langchain.com/docs/modules/chains/foundational/llm_chain
  2. Prompt chains as described in https://js.langchain.com/docs/expression_language/get_started#basic-example-prompt--model--output-parser

Direct chains utilize BaseChain.prototype.invoke to kick off their work. We should instrument this method with an auto-injected BaseCallbackHandler that responds to the correct events.

Prompt chains utilize at least two invoke methods:

  1. BaseChatModel.prototype.invoke when a model in the chain is invoked
  2. BasePromptTemplate.prototype.invoke when the prompt chain is started/ended

These methods should be instrumented with an appropriate callback handler.

In all cases, the corresponding LLM event types should be created and recorded. In these cases, the LLM events should have virtual_llm set to true.

Spans should be named Llm/chain/Langchain/invoke for chains and Llm/agent/Langchain/invoke for models.

This implementation should honor the feature flag config.feature_flag.langchain_instrumentation.

workato-integration[bot] commented 6 months ago

https://new-relic.atlassian.net/browse/NR-218481