search

nomic-ai / gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
https://nomic.ai/gpt4all
MIT License
69.01k stars 7.57k forks source link

RuntimeError: Unable to instantiate model: CPU does not support AVX (M1 Macbook) #2744

Closed kesslerr closed 1 month ago

kesslerr commented 1 month ago

Dear all, I am sorry but i was seeing some related but different issues, but was not really finding an answer that worked (M1 Macbook, 16GB)

Bug Report

When initialising the model as in the example, it says unable because CPU does not support AVX.

Example Code

from gpt4all import GPT4All
model = GPT4All(model_name="orca-mini-3b-gguf2-q4_0.gguf",
                model_path="../models/") # downloads / loads a 4.66GB LLM

Steps to Reproduce

Just followed the instructions from the tutorial. Tested were the following scenarios

  1. Conda environment
  2. Pyenv environment

Wither the current version 2.7.0 of gpt4all, or an older 0.2.3.

Different models:

"Meta-Llama-3-8B-Instruct.Q4_0.gguf" "orca-mini-3b-gguf2-q4_0.gguf"

The problem remains.

Expected Behavior

Should initialize a model.

Your Environment

System Version: macOS 14.5 (23F79) Kernel Version: Darwin 23.5.0 Python 3.12.4

Chip: Apple M1 Total Number of Cores: 8 (4 performance and 4 efficiency) Memory: 16 GB

Environment infos (exemplary pyenvs tested):

Name: gpt4all Version: 2.7.0 Summary: Python bindings for GPT4All Home-page: https://gpt4all.io/ Author: Nomic and the Open Source Community Author-email: support@nomic.ai License: Location: /Users/roman/GitHub/VocabVerse/vocab/lib/python3.12/site-packages Requires: requests, tqdm Required-by:

or

Name: gpt4all Version: 0.2.3 Summary: Python bindings for GPT4All Home-page: https://pypi.org/project/gpt4all/ Author: Richard Guo Author-email: richard@nomic.ai License: Location: /Users/roman/GitHub/VocabVerse/vocab/lib/python3.12/site-packages Requires: requests, tqdm Required-by:

cosmic-snow commented 1 month ago

That's surprising. The bindings should not require AVX on a M1. I don't have one, so I'm unable to look into that, sorry.

cebtenzzre commented 1 month ago

You are using an Intel x86_64 build of Python, which runs in Rosetta and does not support AVX instructions. file "$(which python)" should confirm this, it is supposed to be Mach-O 64-bit executable arm64 on M1.

This is not a supported use of the GPT4All Python binding. Please install in an environment that uses an arm64 build of Python.

cosmic-snow commented 1 month ago

Right, I think that might've even been mentioned in another issue quite some time ago. I didn't remember.

By the way, it could be a good idea to check it (and raise) with one or more of sys.platform, platform.system(), platform.processor(), and/or the sysctl call mentioned here: https://stackoverflow.com/questions/72888632/how-to-check-if-python-is-running-on-an-m1-mac-even-under-rosetta/74654713#74654713

I wouldn't be able to test that myself, though.

kesslerr commented 1 month ago

Hey, thanks a lot, that was the problem... :(

platform.processor() returns indeed 'i386', so it's running in rosetta. Which Python returned Mach-O 64-bit executable x86_64