search

simonw / llm-mlc

LLM plugin for running models using MLC
Apache License 2.0
179 stars 8 forks source link

"Incompatible architecture" errors after installing Llama2 model #17

Open standage opened 11 months ago

standage commented 11 months ago

I followed the following installation process on Intel MacBook Pro (Ventura 13.6) to install llm, llm-mlc, and the Llama2 model.

conda create --name llm -y python=3.11
conda activate llm
pip install llm openai==0.28.1
mamba install -y git-lfs
llm install llm-mlc
pip install --pre -U -f https://mlc.ai/wheels mlc-chat-nightly mlc-ai-nightly
llm mlc setup
llm mlc download-model Llama-2-7b-chat --alias llama2
llm models

But when I try to execute the model, it looks like the installed components are for a different architecture.

$ llm -m Llama-2-7b-chat 'five clever and humorous names for ice cream flavors'
Error: Traceback (most recent call last):
  File "/Users/runner/work/package/package/tvm/src/runtime/dso_library.cc", line 125
InternalError: Check failed: (lib_handle_ != nullptr) is false: Failed to load dynamic shared library dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so dlopen(dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so, 0x0005): tried: 'dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OSdist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file), '/Users/daniel.standage/anaconda3/envs/llm/bin/../lib/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file), '/usr/lib/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file, not in dyld cache), 'dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/Users/daniel.standage/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/daniel.standage/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (no such file), '/Users/daniel.standage/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-metal.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64'))

Did I make a mistake in the installation process? Have I missed something?

ade-times commented 10 months ago

We have the same problem I believe. The rest of the error is

mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')

This means we're trying to build on the wrong type of architecture. There's another issue raised with similar problems. Let me know if you manage to solve this