search

newrelic / node-newrelic

New Relic Node.js agent code base. Developers are welcome to create pull requests here, please see our contributing guidelines. For New Relic technical support, please go to http://support.newrelic.com.
Apache License 2.0
968 stars 397 forks source link

(langchain) Instrument streamed responses #1967

Closed jsumners-nr closed 7 months ago

jsumners-nr commented 8 months ago

LangChain supports streaming responses as described in https://js.langchain.com/docs/modules/model_io/llms/streaming_llm. In short, there are two ways to stream responses:

  1. Through the .stream method on model entities
  2. By settings streaming: true when constructing a model entity and supplying a handleLLMNewToken callback handler.

We will need to instrument both methods. We should be able to instrument base methods as discussed in the non-streaming ticket.

Spans should be named Llm/agent/Langchain/invoke.

workato-integration[bot] commented 8 months ago

https://new-relic.atlassian.net/browse/NR-218483