(langchain): Instrument Bedrock (non-streamed)

[jsumners-nr]

Ideally, any LLMs the LangChain library is capable of interacting with will make use of our existing instrumentation work for those LLMs. This is the case with out OpenAI instrumentation. However, our Bedrock instrumentation is not applicable because LangChain uses a bespoke AWS client. As a result, we need to instrument LangChain's Bedrock implementation.

There are two methods for directly using LLMs with LangChain. LangChain's direct LLM usage for Bedrock is described at https://js.langchain.com/docs/integrations/llms/bedrock and https://js.langchain.com/docs/integrations/chat/bedrock. A resulting instrumented app could look like:

'use strict'

// https://js.langchain.com/docs/integrations/chat/bedrock

require('dotenv').config()
const newrelic = require('newrelic')
// const { BedrockChat: Bedrock } = require('@langchain/community/chat_models/bedrock')
const { Bedrock } = require('@langchain/community/llms/bedrock')

main().then(() => {}).catch(error => {
  console.error(error)
})

async function main () {
  newrelic.startBackgroundTransaction('bedrock-tx', async () => {
    try {
      const chatModel = new Bedrock({
        region: 'us-east-1', // required
        model: 'anthropic.claude-v2' // required
      })
      const res = await chatModel.invoke("what is the answer to life, the universe, and everything?")
      console.log(res)
    } catch (error) {
      console.error(error)
    }

    newrelic.shutdown({ collectPendingData: true }, () => {})
  })
}

Note that the Bedrock import can be interchanged and each is slightly different. For example, the above script will fail unless the llms/bedrock import is swapped for the chat_models/bedrock import. This is because the llms/bedrock invoke will not coerce the given prompt into a proper \n\nHuman: <prompt> format required by the Claude model, but the chat_models/bedrock will. It is unclear if this detail has any bearing on our instrumentation, but it is provided here for clarity.

Both imports utilize BaseChatModel.prototype.invoke as their means of interacting with the LLM. We should be able to instrument this method to cover our needs. Otherwise, we may need to instrument both of:

• https://github.com/langchain-ai/langchainjs/blob/1ea3b3f66ffca6452da9f020838f72db4952ea78/libs/langchain-community/src/chat_models/bedrock/web.ts#L194-L204
• https://github.com/langchain-ai/langchainjs/blob/1ea3b3f66ffca6452da9f020838f72db4952ea78/libs/langchain-community/src/llms/bedrock/web.ts#L119-L129

Spans should be named Llm/agent/Langchain/invoke.

[workato-integration[bot]]

https://new-relic.atlassian.net/browse/NR-218482

[bizob2828]

We're going to track usage first before replicating our instrumentation in langchain for bedrock

[bizob2828]

I have posed a question to see if we can contribute to langchain js instead of adding similar instrumentation for bedrock in langchain.