search

khoj-ai / khoj

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (e.g gpt, claude, gemini, llama, qwen, mistral).
https://khoj.dev
GNU Affero General Public License v3.0
14.25k stars 706 forks source link

Unable to instantiate model #533

Closed ChenZhao44 closed 1 year ago

ChenZhao44 commented 1 year ago

I am not able to load local models on my M1 MacBook Air.

Error messages are as follows.

[11:04:08] INFO     💬 Setting up conversation processor            configure.py:147
100%|█████████████████████████████████████████| 3.79G/3.79G [04:45<00:00, 13.3MiB/s]
Model downloaded at:  /Users/chenzhao/.cache/gpt4all/llama-2-7b-chat.ggmlv3.q4_0.bin
Invalid model file
Found model file at  /Users/chenzhao/.cache/gpt4all/llama-2-7b-chat.ggmlv3.q4_0.bin
Invalid model file
[11:08:58] ERROR    Error while loading offline chat model: Unable to  config.py:112
                    instantiate model
                    ╭────── Traceback (most recent call last) ───────╮
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/khoj/processor/conversatio │
                    │ n/gpt4all/utils.py:16 in download_model        │
                    │                                                │
                    │   13 │                                         │
                    │   14 │   # Use GPU for Chat Model, if availabl │
                    │   15 │   try:                                  │
                    │ ❱ 16 │   │   model = GPT4All(model_name=model_ │
                    │   17 │   │   logger.debug("Loaded chat model t │
                    │   18 │   except ValueError:                    │
                    │   19 │   │   model = GPT4All(model_name=model_ │
                    │                                                │
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/gpt4all/gpt4all.py:97 in   │
                    │ __init__                                       │
                    │                                                │
                    │    94 │   │   self.config: ConfigType = self.r │
                    │       allow_download=allow_download)           │
                    │    95 │   │   if device is not None:           │
                    │    96 │   │   │   if device != "cpu":          │
                    │ ❱  97 │   │   │   │   self.model.init_gpu(mode │
                    │    98 │   │   self.model.load_model(self.confi │
                    │    99 │   │   # Set n_threads                  │
                    │   100 │   │   if n_threads is not None:        │
                    │                                                │
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/gpt4all/pyllmodel.py:225   │
                    │ in init_gpu                                    │
                    │                                                │
                    │   222 │   │   │   model_path_enc = model_path. │
                    │   223 │   │   │   mem_required = llmodel.llmod │
                    │   224 │   │   else:                            │
                    │ ❱ 225 │   │   │   mem_required = self.memory_n │
                    │   226 │   │   device_enc = device.encode("utf- │
                    │   227 │   │   success = self.llmodel_lib.llmod │
                    │       mem_required, device_enc)                │
                    │   228 │   │   if not success:                  │
                    │                                                │
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/gpt4all/pyllmodel.py:192   │
                    │ in memory_needed                               │
                    │                                                │
                    │   189 │   │   if self.model is not None:       │
                    │   190 │   │   │   return llmodel.llmodel_requi │
                    │   191 │   │   else:                            │
                    │ ❱ 192 │   │   │   raise ValueError("Unable to  │
                    │   193 │                                        │
                    │   194 │   def list_gpu(self, model_path: str)  │
                    │   195 │   │   """                              │
                    ╰────────────────────────────────────────────────╯
                    ValueError: Unable to instantiate model

                    During handling of the above exception, another
                    exception occurred:

                    ╭────── Traceback (most recent call last) ───────╮
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/khoj/utils/config.py:108   │
                    │ in __init__                                    │
                    │                                                │
                    │   105 │   │                                    │
                    │   106 │   │   if self.offline_chat.enable_offl │
                    │   107 │   │   │   try:                         │
                    │ ❱ 108 │   │   │   │   self.gpt4all_model.loade │
                    │       download_model(self.offline_chat.chat_mo │
                    │   109 │   │   │   except Exception as e:       │
                    │   110 │   │   │   │   self.offline_chat.enable │
                    │   111 │   │   │   │   self.gpt4all_model.loade │
                    │                                                │
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/khoj/processor/conversatio │
                    │ n/gpt4all/utils.py:19 in download_model        │
                    │                                                │
                    │   16 │   │   model = GPT4All(model_name=model_ │
                    │   17 │   │   logger.debug("Loaded chat model t │
                    │   18 │   except ValueError:                    │
                    │ ❱ 19 │   │   model = GPT4All(model_name=model_ │
                    │   20 │   │   logger.debug("Loaded chat model t │
                    │   21 │                                         │
                    │   22 │   return model                          │
                    │                                                │
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/gpt4all/gpt4all.py:98 in   │
                    │ __init__                                       │
                    │                                                │
                    │    95 │   │   if device is not None:           │
                    │    96 │   │   │   if device != "cpu":          │
                    │    97 │   │   │   │   self.model.init_gpu(mode │
                    │ ❱  98 │   │   self.model.load_model(self.confi │
                    │    99 │   │   # Set n_threads                  │
                    │   100 │   │   if n_threads is not None:        │
                    │   101 │   │   │   self.model.set_thread_count( │
                    │                                                │
                    │ /Users/chenzhao/miniconda3/envs/khoj/lib/pytho │
                    │ n3.11/site-packages/gpt4all/pyllmodel.py:267   │
                    │ in load_model                                  │
                    │                                                │
                    │   264 │   │   if self.model is not None:       │
                    │   265 │   │   │   llmodel.llmodel_loadModel(se │
                    │   266 │   │   else:                            │
                    │ ❱ 267 │   │   │   raise ValueError("Unable to  │
                    │   268 │   │                                    │
                    │   269 │   │   filename = os.path.basename(mode │
                    │   270 │   │   self.model_name = os.path.splite │
                    ╰────────────────────────────────────────────────╯
                    ValueError: Unable to instantiate model
           INFO     127.0.0.1:53817 - "POST                          h11_impl.py:431
                    /api/config/data/processor/conversation/offline_
                    chat?enable_offline_chat=true HTTP/1.1" 200
sabaimran commented 1 year ago

Hi @ChenZhao44 , do you mind sharing what version of Khoj you're on? You can check by running khoj --version. If you're using a pre-release version, you might encounter this error as we've upgraded the GPT4All dependency to use v2.0.0, which isn't compatible with the older llama v2 binary.

If you're on the v0.13.0 release, I would recommend deleting the existing binary rm /Users/chenzhao/.cache/gpt4all/llama-2-7b-chat.ggmlv3.q4_0.bin and restarting Khoj so it downloads again. If you're not on the latest Khoj release, try upgrading like so pip install --upgrade khoj-assistant. That should make the downloads a little more stable.

ChenZhao44 commented 1 year ago

@sabaimran Thanks for your reply. My khoj version is v0.13.0 while the default version of gpt4all installed on my Mac using pip install khoj-assistant is v1.0.12. I was trying to fix it by manually installing the latest version but it was still not working.

I find this issue might be related. It may be an issue with my conda installation that runs in Rosetta mode.

ChenZhao44 commented 1 year ago

This issue was fixed by installing the arm native conda on my Mac.

debanjum commented 1 year ago

Thanks for clarifying that using the ARM maybe Conda fixed the issue for you! It'd be a useful reference for other folks who run into the same issue