search

withcatai / node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. Force a JSON schema on the model output on the generation level
https://withcatai.github.io/node-llama-cpp/
MIT License
735 stars 63 forks source link

illegal hardware instruction on M2 #251

Closed RobertAron closed 1 week ago

RobertAron commented 1 week ago

Issue description

Happy path for using the built in binaries didn't work for me.

Expected Behavior

NPM install chooses the correct binaries for you

Actual Behavior

When I ran 

npx --no node-llama-cpp chat --model ./capybarahermes-2.5-mistral-7b.Q2_K.gguf

I got in response

illegal hardware instruction

Steps to reproduce

Running the first 3 commands on the getting started guide

My Environment

Dependency Version
Operating System macOS Sonoma 14.5
CPU Apple M2 Max
Node.js version v20.3.0
node-llama-cpp version ^2.8.12

Additional Context

Word Around

I was able to work around the problem by using the 

npx --no node-llama-cpp download

command and compiling the binaries myself.

Tiny Suggestion

Perhaps the documentation could be altered slightly to suggest this as a work around in the meantime? If you'd like I can make a PR that does this.

If you'd like the problem to be tackled directly, I'm don't think I have enough context to help. I'm just starting out with learning self hosted LLM stuff so I don't think I have enough context to meaningfully contribute.

Relevant Features Used

Are you willing to resolve this issue by submitting a Pull Request?

No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.

giladgd commented 1 week ago

I couldn't reproduce this issue on my machine (M1 Max).

Can you please run this command and attach the output of it?

npx --yes node-llama-cpp@beta inspect gpu
RobertAron commented 1 week ago
Screenshot 2024-06-28 at 8 11 22 PM 
➜  synthetic-language-translation git:(v1) ✗ npx --yes node-llama-cpp@beta inspect gpu
npm WARN deprecated npmlog@6.0.2: This package is no longer supported.
npm WARN deprecated are-we-there-yet@3.0.1: This package is no longer supported.
npm WARN deprecated gauge@4.0.4: This package is no longer supported.
npm ERR! path /Users/robert/.npm/_npx/f18a9ff1aee9ad13/node_modules/node-llama-cpp
npm ERR! command failed
npm ERR! signal SIGILL
npm ERR! command sh -c node ./dist/cli/cli.js postinstall

npm ERR! A complete log of this run can be found in: /Users/robert/.npm/_logs/2024-06-29T01_10_55_289Z-debug-0.log

I can install it if I do --ignore-scripts. After that I can run it and get this:

OS: macOS 23.5.0 (x64)
Node: 20.3.0 (x64)
TypeScript: 5.5.2
node-llama-cpp: 3.0.0-beta.32

Metal: not supported by llama.cpp on Intel Macs

CPU model: Apple M2 Max
Used RAM: 92.89% (59.46GB/64GB)
Free RAM: 7.1% (4.54GB/64GB)

From here running npx --no node-llama-cpp download gives me the wrong compile. I need to do npx node-llama-cpp download --gpu metal

RobertAron commented 1 week ago

Dug a little more...apparently for the longest time I've been using x64 version of node 🤦

I figured out how to install the arm version and now I get this

npx --yes node-llama-cpp@beta inspect gpu
OS: macOS 23.5.0 (arm64)
Node: 22.3.0 (arm64)
TypeScript: 5.5.2
node-llama-cpp: 3.0.0-beta.32

Metal: available

CPU model: Apple M2 Max
Used RAM: 78.19% (50.04GB/64GB)
Free RAM: 21.8% (13.96GB/64GB)

So I guess this is like 90% user error? You could add guards here but also I think it's reasonable to close this issue.

Thanks for helping me debug! I really appreciate it!

giladgd commented 1 week ago

Detecting that the process is running through a virtualization layer isn't that simple, as far as I've seen, so I don't see how to add guards for this case. On that note, if you have a way to detect that the machine is arm64 while the process is x64 from inside of a x64 nodejs process then let me know.

In the documentation of the upcoming version 3 stable release I'll make sure to mention checking for the arch of the installed nodejs as a troubleshooting measure for crashes.