BigDL-LLM cannot run on MacPro that uses Xeon W CPU

[winchillwest]

It appears that there is binaries for MacOS, which cause the issue.

BigDL/python/llm/example/CPU/HF-Transformers-AutoModels/Model/aquila] python generate.py --prompt "what is AI?"

****Usage Error**** Failed to load shared library '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so': dlopen(/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so, 0x0006): tried: '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (no such file), '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (not a mach-o file). 2023-10-11 17:38:46,089 - ERROR -

****Call Stack***** Traceback (most recent call last): File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/ggml/model/llama/llama_cpp.py", line 98, in _load_shared_library return ctypes.CDLL(str(_lib_path), **cdll_args) File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/init.py", line 366, in init self._handle = _dlopen(self._name, mode) OSError: dlopen(/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so, 0x0006): tried: '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (no such file), '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (not a mach-o file)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/localadmin/Work/LLM/BigDL/python/llm/example/CPU/HF-Transformers-AutoModels/Model/aquila/generate.py", line 43, in model = AutoModelForCausalLM.from_pretrained(model_path, File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/transformers/model.py", line 95, in from_pretrained model = cls.load_convert(q_k, optimize_model, *args, *kwargs) File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/transformers/model.py", line 125, in load_convert model = ggml_convert_low_bit(model, qtype, optimize_model, File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/transformers/convert.py", line 128, in ggml_convert_low_bit model, has_been_replaced = _replace_with_low_bit_linear( File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/transformers/convert.py", line 50, in _replace_with_low_bit_linear from bigdl.llm.transformers.low_bit_linear import LowBitLinear, FP4Params, FP16Linear File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/transformers/low_bit_linear.py", line 55, in import bigdl.llm.ggml.model.llama.llama_cpp as ggml File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/ggml/model/llama/init.py", line 22, in from .llama_cpp import File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/ggml/model/llama/llama_cpp.py", line 109, in _lib = _load_shared_library(_lib_base_name) File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/ggml/model/llama/llama_cpp.py", line 100, in _load_shared_library invalidInputError(False, f"Failed to load shared library '{_lib_path}': {e}.") File "/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/utils/common/log4Error.py", line 32, in invalidInputError raise RuntimeError(errMsg) RuntimeError: Failed to load shared library '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so': dlopen(/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so, 0x0006): tried: '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (no such file), '/Users/localadmin/py3-venv/lib/python3.9/site-packages/bigdl/llm/libs/libllama-api.so' (not a mach-o file).

[hkvision]

Hi @winchillwest

We don't support MacOS actually and we only release for Linux/Windows. You are highly recommended to run on Linux/Windows instead.

[jason-dai]

@winchillwest you may now try running bigdl-llm docker on MacOS (with Intel CPU): https://github.com/intel-analytics/BigDL/tree/main/docker/llm