Getting "illegal hardware instruction" error running on M1 MacBook Pro

[nini1294]

When I download and run the llamafile from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4-server.llamafile\?download\=true I get an error like below.

[1] 96405 illegal hardware instruction ./llamafile

Running the executable with the --help option seems to work fine

[jart]

Could you disable SIP and run lldb -- $TMPDIR/.ape-1.8 ./llava-v1.5-7b-q4-server.llamafile and give me (1) the name of the instruction that's illegal (or its hex value) and (2) the hex address of where that instruction is in memory?

[nini1294]

Hi, not too sure if I'm doing it correctly but here are the steps and output.

1. Had to modify from $TMPDIR/.ape-1.8 to $TMPDIR/.ape-1.9 since .ape-1.8 was not found
2. Once lldb was running, I gave the run command

Output from the run step

(lldb) run
Process 97359 launched: '/var/folders/5t/nwvcm6313sq3vfycrj_6rcz00000gn/T/.ape-1.9' (x86_64)
Process 97359 stopped
* thread #1, stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00000000004cfea2
->  0x4cfea2: vmovd  %eax, %xmm5
    0x4cfea6: vsubss 0x1315de(%rip), %xmm5, %xmm4
    0x4cfeae: vmovd  %xmm4, %eax
    0x4cfeb2: orl    %eax, %ecx
Target 0: (.ape-1.9) stopped.

Let me know if I should try some other/additional steps

[jart]

That's really strange that it's showing x86 code. Is this running in rosetta somehow? What shows up if you say:

file $TMPDIR/.ape-1.9

[nini1294]

This is what I get from that command /var/folders/5t/nwvcm6313sq3vfycrj_6rcz00000gn/T//.ape-1.9: Mach-O 64-bit executable x86_64

[jart]

What's the output of uname -m on your system?

[nini1294]

Output from uname -m x86_64

Thats odd

[sagarjauhari]

Getting a 'killed' error on M1 MacPro. Not sure if it's the same issue:

~/code/llama_llava
❯ ./llamafile-server-0.1-llava-v1.5-7b-q4
zsh: killed     ./llamafile-server-0.1-llava-v1.5-7b-q4

~/code/llama_llava
❯ lldb -- $TMPDIR/.ape-1.9 ./llamafile-server-0.1-llava-v1.5-7b-q4
(lldb) target create "/var/folders/y2/5j_s4h6j3b1fcvqfq6wxq7g80000gp/T//.ape-1.9"
Current executable set to '/var/folders/y2/5j_s4h6j3b1fcvqfq6wxq7g80000gp/T/.ape-1.9' (arm64).
(lldb) settings set -- target.run-args  "./llamafile-server-0.1-llava-v1.5-7b-q4"
(lldb) quit

~/code/llama_llava
❯ uname -m
arm64

~/code/llama_llava
❯ sh -C ./llamafile-server-0.1-llava-v1.5-7b-q4
zsh: killed     sh -C ./llamafile-server-0.1-llava-v1.5-7b-q4

[andreiHaz]

I encountered the same issue as the original poster. Here are the steps that resolved it for me:

1. Check iTerm Settings:

   • Ensure that iTerm (or your terminal application) is not running with the "Open using Rosetta" option. You can verify this by right-clicking on the iTerm app in Finder, selecting "Get Info", and making sure "Open using Rosetta" is unchecked.

2. Remove the Temporary File:

   • Delete the .ape-1.9 file from the temporary directory. This can be done by executing the following command in the terminal:

     rm $TMPDIR/.ape-1.9

3. Run the Executable:

   • After performing the above steps, run the llamafile-server executable with the following command:

     ./llamafile-server-0.1-llava-v1.5-7b-q4

   • This should work without any issues now.

These steps successfully resolved the problem for me. Hope this helps!

[jart]

@sagarjauhari the Killed error turned out to be CrowdStrike. Ask to get whitelisted. See #14

[jart]

@nini1294 You must be running inside some kind of Rosetta shell environment, or possibly an x86 Docker container. Try opening Terminal.app and running llamafile on the standard system shell. uname -m should report arm64 if you have an M1 or M2 or M3 microprocessor. I'm going to close this issue out for now. If you determine further details about how your x86 environment happened, then please feel free to share it here for future readers.

[nini1294]

Hi, thanks a lot for the input, you were right that it was somehow due to my shell being in x86_64 rather than arm64.

I checked and iTerm and the Terminal app both were running as native, but somehow all the my shells seemed to be starting in x86_64 so all the processes started under them were as well.

What I ended up doing was to temporarily force start a zsh instance to start in arm64 using:

env /usr/bin/arch -arm64 /bin/zsh --login

That seemed to do the trick and I have llamafile running now!

Thanks again!

[jart]

Thanks for circling back and sharing that @nini1294. Glad to hear llamafile is working for you!

[derwiki]

That's really strange that it's showing x86 code. Is this running in rosetta somehow? What shows up if you say:

file $TMPDIR/.ape-1.9

FWIW my .ape-1.9 was showing:

➜  llava ./llamafile-server-0.1-llava-v1.5-7b-q4 
[1]    24997 illegal hardware instruction  ./llamafile-server-0.1-llava-v1.5-7b-q4
➜  llava file $TMPDIR/.ape-1.9
/var/folders/j2/ms4kyj991ps6k_dmls2jlxnw0000gn/T//.ape-1.9: Mach-O 64-bit executable x86_64
➜  llava uname -m
arm64

Per @andreiHaz I removed .ape-1.9

➜  llava rm $TMPDIR/.ape-1.9

And now it boots up as expected! ✨ Thanks everyone!