Hardware incompatibility error

[venuv]

Hi, I installed mlc_chat on metal on my Macbook Pro (M2 Max). Assume the hardware incompatibility error is that the standard llama models are x86. If so where can I find equivalent (e.g. to llama-2-13v-chat) llama models that are arm64 compatible?

llm -m mlc-chat-Llama-2-7b-chat-hf-q4f16_1 'five common male first names' Error: Traceback (most recent call last): File "/Users/runner/work/package/package/tvm/src/runtime/dso_library.cc", line 125 InternalError: Check failed: (libhandle != nullptr) is false: Failed to load dynamic shared library /Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so dlopen(/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so, 0x0005): tried: '/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file), '/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

[simonw]

This is surprising - I use an M2 Mac myself and everything is working correctly.

How did you install LLM? Which Python version are you using?

Try running this and see what it says:

python -c 'import sys; print(sys.version)'

[venuv]

(llmcli) R77NK6JXG7:projects venuvasudevan$ python -c 'import sys; print(sys.version)' 3.9.5 (default, May 18 2021, 12:31:01) [Clang 10.0.0 ]

[venuv]

@simonw I reinstalled both llm and llm-mlc on 3.11.4 .. seems to end up in the same place

(llmcli) R77NK6JXG7:llmcli venuvasudevan$ pip list|grep llm-mlc llm-mlc 0.4 (llmcli) R77NK6JXG7:llmcli venuvasudevan$ llm mlc models MlcModel: mlc-chat-Llama-2-7b-chat-hf-q4f16_1 (aliases: llama2, Llama-2-7b-chat) (llmcli) R77NK6JXG7:llmcli venuvasudevan$ llm -m Llama-2-7b-chat 'five names for a cute pet ferret' Error: Traceback (most recent call last): File "/Users/runner/work/package/package/tvm/src/runtime/dso_library.cc", line 125 InternalError: Check failed: (libhandle != nullptr) is false: Failed to load dynamic shared library /Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so dlopen(/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so, 0x0005): tried: '/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file), '/Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')) (llmcli) R77NK6JXG7:llmcli venuvasudevan$

[venuv]

@simonw I think tvm is looking for the file - /Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so. In my system that file is in a 'backslashed' directory at - /Users/venuvasudevan/Library/Application\ Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so.

lmk if there is a settable parameter

---

(llmcli) R77NK6JXG7:llm-mlc venuvasudevan$ ls /Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so ls: /Users/venuvasudevan/Library/Application: No such file or directory ls: Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so: No such file or directory (llmcli) R77NK6JXG7:llm-mlc venuvasudevan$ ls /Users/venuvasudevan/Library/Application\ Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so /Users/venuvasudevan/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so

[avivo]

I had the same error on an M2 air. LLM installed via brew. That said, this machine has perhaps a non-ideal python setup, which I no longer remember. It's been a long time since I have written a lot of code and I'm usually able to work with whatever happens.

Error: Traceback (most recent call last): File "/Users/runner/work/package/package/tvm/src/runtime/dso_library.cc", line 125 InternalError: Check failed: (libhandle != nullptr) is false: Failed to load dynamic shared library /Users/av/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so dlopen(/Users/av/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so, 0x0005): tried: '/Users/av/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/aviv/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file), '/Users/av/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

python -c 'import sys; print(sys.version)'
pyenv: python: command not found

The `python' command exists in these Python versions:
  anaconda3-2022.05
  anaconda3-2022.05/envs/invokeai

python3 -c 'import sys; print(sys.version)'
3.11.4 (main, Jul 25 2023, 17:07:07) [Clang 14.0.3 (clang-1403.0.22.14.1)]

[venuv]

indeed @avivo - one final documentation of the M2 Max strangeness - mlc_chat CLI works (because it reaches for the metal_x86_64), python version doesn't (with error similar to above)

[kugg]

The problem is caused by using Python interpreters compiled for different architectures. Make sure that your Python interpreter is running as arm64 or arm64e.

file $(which python3)

Here are two desirable architectures:

/Users/XXX/miniforge3/bin/python3.10: Mach-O 64-bit executable arm64
/usr/bin/python3 (for architecture arm64e):     Mach-O 64-bit executable arm64e

When executables are compiled with "multiple architectures", the arch command can be used to select which one to use.

arch -arm64e /usr/bin/python3

I wrote a more lengthy explanation here: https://github.com/abetlen/llama-cpp-python/issues/847#issuecomment-1786083870

[ade-times]

@kugg I'm using a mac intel machine but getting the same error. It seems to be pulling the metal.so library file which is arm64 architecture. But I need the x86_64 one. I'm not sure how to get around this. Is the x86 file in the mlc library as well? if so which one is it: https://github.com/mlc-ai/binary-mlc-llm-libs Or is there a different way to address this

Thanks!

[ade-times]

Also to add @simonw the mlc repo has a tutorial where this is set: cm = ChatModule(model="dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1", lib_path="dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so") Is that something the llm-mlc package does somewhere. I'm thinking if we can point to a different .so file we may be out of the woods here. But that's just a guess from me

[ade-times]

Further update! it does work with the mlc_chat CLI and points correctly it seems to the correct dylib file which is in the llm-mlc package!