LlamaCpp Model callbacks handleLLMNewToken JS/TS

[jffry93]

It doesnt looks like I am able to use handleLLMNewToken with the LlamaCpp model. Can someone assist me with detokenizing the token from my onToken function?

    const model = new LlamaCpp({ modelPath: llamaPath });
    const response = await model.call(question, {
      onToken: (token) => {
        console.log(token); // ===  number[]
      },
    });

Or can someone guide me how to use handleLLMNewToken with the LlamaCpp ?

[dosubot[bot]]

🤖

Hello,

To use the handleLLMNewToken function with the LlamaCpp model to detokenize the token from your onToken function in the LangChain framework, you need to override the handleLLMNewToken method in a class that extends BaseCallbackHandler.

Here is an example of how you can do this:

import { BaseCallbackHandler, NewTokenIndices, HandleLLMNewTokenCallbackFields } from 'langchain/src/callbacks/base.ts';

class MyCallbackHandler extends BaseCallbackHandler {
  name = 'MyCallbackHandler';

  async handleLLMNewToken(
    token: string,
    idx: NewTokenIndices,
    runId: string,
    parentRunId?: string,
    tags?: string[],
    fields?: HandleLLMNewTokenCallbackFields
  ) {
    // Here you can use your LlamaCpp model to detokenize the token
    const detokenized = await myLlamaCppModel.detokenize(token);
    console.log(`Detokenized token: ${detokenized}`);
  }
}

// Then you can use your custom callback handler in your LangChain application
const myCallbackHandler = new MyCallbackHandler();

In this example, myLlamaCppModel is a placeholder for your LlamaCpp model instance. You would replace this with the actual instance of your model. The detokenize method is also a placeholder and should be replaced with the actual method used to detokenize tokens in your model.

Please note that the handleLLMNewToken method is called when an LLM/ChatModel in streaming mode produces a new token. The token parameter is the new token produced by the model. The idx parameter is an object with the indices of the prompt and completion that produced the token. The runId parameter is the ID of the run that produced the token. The parentRunId parameter is the ID of the parent run, if any. The tags parameter is an array of tags associated with the run. The fields parameter is an object with additional callback fields.

In the context shared, the handleLLMNewToken function is used in a callback passed to the call method of an LLMChain instance. This is another way to use the handleLLMNewToken function in the LangChain framework. However, the function in this context is not detokenizing the token with a LlamaCpp model, but simply writing the token to the standard output.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• langchain/src/callbacks/base.ts
• examples/src/chains/llm_chain_stream.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

Hi, @jffry93,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. From what I understand, you were having difficulty using the handleLLMNewToken with the LlamaCpp model in JavaScript/TypeScript and were seeking assistance with detokenizing the token from the onToken function or guidance on using handleLLMNewToken with LlamaCpp. I provided a detailed response with an example of how to override the handleLLMNewToken method in a class that extends BaseCallbackHandler, including code snippets and explanations on how to detokenize the token using the LlamaCpp model.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you!