[Question] How to use transformer.js in langchain

[mrddter]

Hi all, I'm writing a custom LLM to use transformer.js with langchain. Does a structure like this make sense? Any advice for optimizing it or best practices to apply?

Any suggestions or feedback would be greatly appreciated 😊 🚀

import { pipeline } from "@xenova/transformers";
import { LLM } from "langchain/llms/base";

class MyHF extends LLM {
  static instance = null;

  constructor(modelTask = "text2text-generation", modelName = "Xenova/LaMini-Flan-T5-783M") {
    super({ maxConcurrency: 1 });
    this.modelTask = modelTask;
    this.modelName = modelName;
    this.llmModel = MyHF.getInstance(this.modelTask, this.modelName);
  }

  static async getInstance(modelTask, modelName, progress_callback = null) {
    if (this.instance === null) {
      this.instance = pipeline(modelTask, modelName, { progress_callback });
    }
    return this.instance;
  }

  _llmType() {
    return "hf";
  }

  async _call(prompt, options = { topk: 1 }) {
    const executor = await MyHF.getInstance(this.modelTask, this.modelName);
    const { generated_text } = await executor(prompt, options);
    return generated_text
  }
}

export default MyHF;

[xenova]

Hi there 👋 Sorry for the late reply :)

Your usage looks correct, and adopts the advised "Singleton" pattern to avoid multiple reconstructions of the pipeline. Just a question about where you intend on running this: in-browser or server-side? If server-side, you can ignore what I will say, but if in-browser, it's usually advised to either use a web worker, or to use onnxruntime-web's proxy option.

[dmmagdal]

@mrddter Did your implementation end up working?