Local generation crashes with ERROR:crashpad_client_win.cc(867)] not connected

[pattontim]

Describe the bug When I try to generate a response using a local model the app immediately crashes without error message.

Terminal:

Indexing files in directory
config is:  { type: 'repo', repoName: 'Xenova/bge-base-en-v1.5' }
getting files info list
getting generation params undefined
hardwareConfig: { useGPU: true, useCUDA: true, useVulkan: false }
[2556:0317/005031.970:ERROR:crashpad_client_win.cc(867)] not connected

To Reproduce Steps to reproduce the behavior:

1. Launch app
2. Ensure the GPU and CUDA are enabled and a model is selected.
3. Click refresh chat icon OR type in "hello" and hit
4. The app crashes

Expected behavior The chat completion finishes with response.

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
  • Hardware: AMD FX, RTX 3060 12GB, 28 GB of RAM
• Version 0.1.55 exe from releases

Additional context Attempted using mixtral 26GB model but issue also presents using the smallest, openhermes-2.5-mistral-7b.Q2_K.gguf (3 GB) Issue presents even if GPU is disabled.

[pattontim]

Same issue using the dev branch.

It crashes on this call:

const newSessionId = await window.llm.createSession(
        "some_unique_session_id"
      );

This looks like potentially an issue with node-llama-cpp stable and beta, since if I attempt to load a chat using node-llama-cpp on the command line it loads the model successfully but exits without any fail reason on a fresh install.

[samlhuillier]

Yes unfortunately we've been having several problems with node llama cpp and have moved entirely towards using Ollama: #135. This should give us much better stability and support across different machines. We will merge and release in the next day or so!

[samlhuillier]

@pattontim the main branch now fully uses Ollama to run local models!

It'd be great if you could test it and let me know whether it works for you.

In Settings->LLM->Add New Local LLM you will need to paste in the model name from the Ollama library you want to use and Reor will download and attach it for you.