Error: ENOENT: no such file or directory, open undefinedbinariesGithubRelease.json

[linonetwo]

Issue description

Throw error when try to load model, seems it try to read a json

Expected Behavior

I'm bundling this in a Tiddlywiki nodejs electron app, and I bundle the prebuild binary with it, so it should not try to get anything from Github.

Actual Behavior

It throw Error: ENOENT: no such file or directory, open 'C:\\Users\\linonetwo\\Documents\\repo-c\\TidGi-Desktop\\undefinedbinariesGithubRelease.json'

I think it comes from https://github.com/withcatai/node-llama-cpp/blob/6b012a6d431d7025737f4fbbfad70e232624dc2b/src/config.ts#L23

Steps to reproduce

My usage is

let llamaInstance: undefined | Llama;
let modelInstance: undefined | LlamaModel;
let contextInstance: undefined | LlamaContext;
let contextSequenceInstance: undefined | LlamaContextSequence;
export async function loadLLamaAndModel(
  loadConfigOverwrite: Partial<LlamaModelOptions> & Pick<LlamaModelOptions, 'modelPath'>,
  conversationID: string,
  subscriber: Subscriber<ILanguageModelWorkerResponse>,
) {
  const loggerCommonMeta = { level: 'info' as const, meta: { function: 'llmWorker.loadLLama' }, id: 'loadLLama' };
  // TODO: maybe use dynamic import cjs version to fix https://github.com/andywer/threads.js/issues/478 ? If get `Timeout: Did not receive an init message from worker`.
  // subscriber.next({ message: 'async importing library', ...loggerCommonMeta });
  subscriber.next({ message: 'library loaded, new LLM now', ...loggerCommonMeta });
  try {
    llamaInstance = await getLlama({
      skipDownload: true,
      build: 'never',
      vramPadding: 0,
      logger: (level, message) => {
        subscriber.next({ message, ...loggerCommonMeta });
      },
    });
    subscriber.next({ message: 'prepared to load model', ...loggerCommonMeta, meta: { ...loggerCommonMeta.meta, loadConfigOverwrite } });
    const onLoadProgress = debounce((percentage: number) => {
      subscriber.next({
        type: 'progress',
        percentage,
        id: conversationID,
      });
    });
    const loadConfig: LlamaModelOptions = {
      onLoadProgress,
      ...loadConfigOverwrite,
    };
    modelInstance = await llamaInstance.loadModel(loadConfig);
    subscriber.next({ message: 'instance loaded', ...loggerCommonMeta });
    return modelInstance;
  } catch (error) {
    await unloadLLama();
    throw error;
  }
}
export async function unloadLLama() {
  await contextInstance?.dispose();
  contextSequenceInstance?.dispose();
  await modelInstance?.dispose();
  await llamaInstance?.dispose();
  contextSequenceInstance = undefined;
  llamaInstance = undefined;
  modelInstance = undefined;
}

My Environment

Dependency	Version
Operating System	Win11
CPU	Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz 2.90 GHz
Typescript version	5.4.2
node-llama-cpp version	3.0.0-beta.15

Electron Version: 29.2.0 Node Version: 20.9.0 Chromium Version: 122.0.6261.156

Additional Context

No response

Relevant Features Used

• [ ] Metal support
• [ ] CUDA support
• [ ] Grammar

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, and I know how to start.

[linonetwo]

This cause __dirname to be undefined

https://github.com/withcatai/node-llama-cpp/blob/6b012a6d431d7025737f4fbbfad70e232624dc2b/src/config.ts#L9C1-L9C64

[linonetwo]

This fix it on userland

  new webpack.DefinePlugin({
    'process.env.NODE_ENV': `"${process.env.NODE_ENV ?? 'production'}"`,
    'const __dirname = path.dirname(fileURLToPath(import.meta.url));': '',
  }),

[linonetwo]

path.join(__dirname, "..", "llama") still have bug

__dirname C:\Users\linonetwo\Documents\repo-c\TidGi-Desktop\node_modules\.pnpm\node-llama-cpp@3.0.0-beta.15_typescript@5.4.2\node_modules\node-llama-cpp\dist
llamaDirectory undefinedllama
binariesGithubReleasePath undefinedbinariesGithubRelease.json

Difficult to find why, sometimes it is good

llamaPrebuiltBinsDirectory C:\Users\linonetwo\Documents\repo-c\TidGi-Desktop\.webpack\main/native_modules/llamaBins win-x64-vulkan
buildMetadataFilePath C:\Users\linonetwo\Documents\repo-c\TidGi-Desktop\.webpack\main\native_modules\llamaBins\win-x64-vulkan\_nlcBuildMetadata.json
prebuiltBinPath C:\Users\linonetwo\Documents\repo-c\TidGi-Desktop\.webpack\main\native_modules\llamaBins\win-x64-vulkan\llama-addon.node

sometimes bad

llamaPrebuiltBinsDirectory undefinedllamaBins win-x64-vulkan
buildMetadataFilePath undefinedllamaBins\win-x64-vulkan\_nlcBuildMetadata.json
prebuiltBinPath null

[linonetwo]

How to avoid using things in this folder? I already bundle the prebuild binary within electron, can I just get rid of /llama folder? And assign a custom path to /llamaBin folder, so I can handle the path myself after electron bundled.

[linonetwo]

It is acturally called by this line, have to patch to comment out this.

https://github.com/withcatai/node-llama-cpp/blob/7878c8a7aa3a44e8f74962bbfae558ac53538c4e/src/config.ts#L38

[giladgd]

@linonetwo The problem in your project is that you're trying to transpile node-llama-cpp to CommonJS, although node-llama-cpp is an ESM-only module. You should mark node-llama-cpp in Webpack as an external module so it won't be bundled together with other code and be left in its original structure, and then await import("node-llama-cpp") in your code to use it.

Modifying and moving around internal files of any module is inadvisable as it leads to instability, and maintaining support for future updates is impractical this way. I plan to do #89 in the near future to provide good examples of how node-llama-cpp can be used in various types of projects; I'll make sure to add an Electron example.

[linonetwo]

@giladgd Thanks. But to external this, usually it will have to also add dependency to resources, like

https://github.com/tiddly-gittly/TidGi-Desktop/blob/16949cdf0c4dc7f32e3909c58b7c5f9ab6186336/scripts/afterPack.js#L61-L91

And node-llama-cpp have lots of dependency, this will make electron resource folder very big.

Besides, will you consider allow me upstream my patch, that disable the initial call to read JSON file, and read path from global variable?

[giladgd]

@linonetwo The dependencies of node-llama-cpp also cannot be bundled if you want to provide a consistent and stable experience.

Bundling won't necessarily save you storage space, it'll just combine many files together so your output will be fewer files. This approach is useful on the web where you want to minimize the number of files a client downloads so the page will load faster, but on an app installed locally on a user machine, the benefits of doing this are insignificant.

Furthermore, while npm modules made for the web are designed with the ability to do tree-shaking, minimization and bundling on them in mind, node modules are not, and native modules in particular rely on the structure of the files they're shipped with. While minimizing bundled code can save some storage space, the savings won't make a noticeable difference in an Electron app's runtime performance.

Electron has its own bundling solution that is designed to work with node modules by taking a different approach that keeps the files structure as-is, but combines all the files into a single .asar archive that compressed the code, this way you can achieve both a smaller installation file, take less disk space when installed, and make sure all node modules work like you expect them to and not bother with the internals of node modules when using Webpack.

You should only use Webpack for the frontend side of Electron, not for the node side.

[linonetwo]

Oh, I thought .node files won't work inside .asar, so I always copy them to resource folder use afterBuild script. I will try if this works.

You should only use Webpack for the frontend side of Electron, not for the node side.

I'm using electron forge, and it bundles both main and renderer, so I have to do it this way...Anyway the workaround works, I'm going to focus on learning the ChatWrapper, and deal with the bundle problems later, (or wait for your electron example) thank you!