Use Claude3 LLM

[eric-gardyn]

I wanted to test with Claude3 LLM. Out-of-the-box, framework didn't work; just wanted to share my changes:

I had to change LangchainChatLlm.ts for answerQuestionAwaited, with

...
      const prompts = ChatPromptTemplate.fromMessages(
        messages.map(m => messageBaseMessagePromptTemplateLike(m))
      )
      const chain = prompts.pipe(chatModel)
      const res = await chain.invoke({})
...

function messageBaseMessagePromptTemplateLike(
  message: OpenAiChatMessage
): BaseMessagePromptTemplateLike {
  return [message.role, message.content ?? '']
}

[mongodben]

thanks eric. i also want to test with claude 3, so will keep this in mind 😄

are you using claude from anthropic directly or another provider like bedrock?

[eric-gardyn]

directly from Anthropic; this is just for a comparison test

[mongodben]

got it, will help me repro the issue.

maybe while at it, will try it out in our eval suite to see comparison

[eric-gardyn]

btw, is the eval suite ready to be used? I am curious to try it

[mongodben]

yes it's available on npm and works, but the docs need expansion https://mongodb.github.io/chatbot/evaluation/. i'm planning on working on this over the next week or so.

our eval suite can be found here - https://github.com/mongodb/chatbot/tree/main/packages/chatbot-eval-mongodb-public

copying the patterns from here is probably the quickest way to get started.

let me know if you'd like to do a call to get set up onboarded w/ it. you can reach out on linkedin for that.