RuntimeError: Failed to import transformers.models.llama.tokenization_llama_fast

[cryptoB0T]

runtime error

RuntimeError: Failed to import transformers.models.llama.tokenization_llama_fast because of the following error
(look up to see its traceback):
tokenizers>=0.13.3 is required for a normal functioning of this module, but found tokenizers==0.13.2.

i had it running previously weeks ago, but at some point it stopped working. i uninstalled and reinstalled the pkg python3 -m pip uninstall khoj-assistant && python3 -m pip install khoj-assistant

here's the full log below:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/lib/python3.11/site-packages/transformers/utils/import_utils.py:1146 in            │
│ _get_module                                                                                      │
│                                                                                                  │
│   1143 │                                                                                         │
│   1144 │   def _get_module(self, module_name: str):                                              │
│   1145 │   │   try:                                                                              │
│ ❱ 1146 │   │   │   return importlib.import_module("." + module_name, self.__name__)              │
│   1147 │   │   except Exception as e:                                                            │
│   1148 │   │   │   raise RuntimeError(                                                           │
│   1149 │   │   │   │   f"Failed to import {self.__name__}.{module_name} because of the followin  │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3. │
│ 11/importlib/__init__.py:126 in import_module                                                    │
│                                                                                                  │
│   123 │   │   │   if character != '.':                                                           │
│   124 │   │   │   │   break                                                                      │
│   125 │   │   │   level += 1                                                                     │
│ ❱ 126 │   return _bootstrap._gcd_import(name[level:], package, level)                            │
│   127                                                                                            │
│   128                                                                                            │
│   129 _RELOADING = {}                                                                            │
│ in _gcd_import:1204                                                                              │
│ in _find_and_load:1176                                                                           │
│ in _find_and_load_unlocked:1147                                                                  │
│ in _load_unlocked:690                                                                            │
│ in exec_module:940                                                                               │
│ in _call_with_frames_removed:241                                                                 │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama_fast.py: │
│ 24 in <module>                                                                                   │
│                                                                                                  │
│    21 from ...utils.versions import require_version                                              │
│    22                                                                                            │
│    23                                                                                            │
│ ❱  24 require_version("tokenizers>=0.13.3")                                                      │
│    25                                                                                            │
│    26 if is_sentencepiece_available():                                                           │
│    27 │   from .tokenization_llama import LlamaTokenizer                                         │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/transformers/utils/versions.py:117 in require_version │
│                                                                                                  │
│   114 │   # check that the right version is installed if version number or a range was provide   │
│   115 │   if want_ver is not None:                                                               │
│   116 │   │   for op, want_ver in wanted.items():                                                │
│ ❱ 117 │   │   │   _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)               │
│   118                                                                                            │
│   119                                                                                            │
│   120 def require_version_core(requirement):                                                     │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/transformers/utils/versions.py:50 in                  │
│ _compare_versions                                                                                │
│                                                                                                  │
│    47 │   │   │   f" reinstalling {pkg}."                                                        │
│    48 │   │   )                                                                                  │
│    49 │   if not ops[op](version.parse(got_ver), version.parse(want_ver)):                       │
│ ❱  50 │   │   raise ImportError(                                                                 │
│    51 │   │   │   f"{requirement} is required for a normal functioning of this module, but fou   │
│    52 │   │   )                                                                                  │
│    53                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: tokenizers>=0.13.3 is required for a normal functioning of this module, but found tokenizers==0.13.2.

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/bin/khoj:8 in <module>                                                             │
│                                                                                                  │
│   5 from khoj.main import run                                                                    │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(run())                                                                          │
│   9                                                                                              │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/main.py:79 in run                                │
│                                                                                                  │
│    76 │   │                                                                                      │
│    77 │   │   # Start Server                                                                     │
│    78 │   │   initialize_server(args.config, args.regenerate, required=False)                    │
│ ❱  79 │   │   configure_routes(app)                                                              │
│    80 │   │   start_server(app, host=args.host, port=args.port, socket=args.socket)              │
│    81 │   else:                                                                                  │
│    82 │   │   from PySide6 import QtWidgets                                                      │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/configure.py:99 in configure_routes              │
│                                                                                                  │
│    96                                                                                            │
│    97 def configure_routes(app):                                                                 │
│    98 │   # Import APIs here to setup search types before while configuring server               │
│ ❱  99 │   from khoj.routers.api import api                                                       │
│   100 │   from khoj.routers.api_beta import api_beta                                             │
│   101 │   from khoj.routers.web_client import web_client                                         │
│   102                                                                                            │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/routers/api.py:39 in <module>                    │
│                                                                                                  │
│    36 from khoj.utils import state, constants                                                    │
│    37 from khoj.utils.yaml import save_config_to_file_updated_state                              │
│    38 from fastapi.responses import StreamingResponse, Response                                  │
│ ❱  39 from khoj.routers.helpers import (                                                         │
│    40 │   get_conversation_command,                                                              │
│    41 │   perform_chat_checks,                                                                   │
│    42 │   generate_chat_response,                                                                │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/routers/helpers.py:10 in <module>                │
│                                                                                                  │
│     7                                                                                            │
│     8 from khoj.utils import state                                                               │
│     9 from khoj.utils.helpers import ConversationCommand, timer, log_telemetry                   │
│ ❱  10 from khoj.processor.conversation.openai.gpt import converse                                │
│    11 from khoj.processor.conversation.gpt4all.chat_model import converse_offline                │
│    12 from khoj.processor.conversation.utils import reciprocal_conversation_to_chatml, message   │
│    13                                                                                            │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/processor/conversation/openai/gpt.py:12 in       │
│ <module>                                                                                         │
│                                                                                                  │
│     9 # Internal Packages                                                                        │
│    10 from khoj.utils.constants import empty_escape_sequences                                    │
│    11 from khoj.processor.conversation import prompts                                            │
│ ❱  12 from khoj.processor.conversation.openai.utils import (                                     │
│    13 │   chat_completion_with_backoff,                                                          │
│    14 │   completion_with_backoff,                                                               │
│    15 )                                                                                          │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/processor/conversation/openai/utils.py:22 in     │
│ <module>                                                                                         │
│                                                                                                  │
│    19 )                                                                                          │
│    20                                                                                            │
│    21 # Internal Packages                                                                        │
│ ❱  22 from khoj.processor.conversation.utils import ThreadedGenerator                            │
│    23                                                                                            │
│    24                                                                                            │
│    25 logger = logging.getLogger(__name__)                                                       │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/khoj/processor/conversation/utils.py:10 in <module>   │
│                                                                                                  │
│     7                                                                                            │
│     8 # External packages                                                                        │
│     9 from langchain.schema import ChatMessage                                                   │
│ ❱  10 from transformers import LlamaTokenizerFast                                                │
│    11                                                                                            │
│    12 # Internal Packages                                                                        │
│    13 import queue                                                                               │
│ in _handle_fromlist:1229                                                                         │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/transformers/utils/import_utils.py:1137 in            │
│ __getattr__                                                                                      │
│                                                                                                  │
│   1134 │   │   │   value = self._get_module(name)                                                │
│   1135 │   │   elif name in self._class_to_module.keys():                                        │
│   1136 │   │   │   module = self._get_module(self._class_to_module[name])                        │
│ ❱ 1137 │   │   │   value = getattr(module, name)                                                 │
│   1138 │   │   else:                                                                             │
│   1139 │   │   │   raise AttributeError(f"module {self.__name__} has no attribute {name}")       │
│   1140                                                                                           │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/transformers/utils/import_utils.py:1136 in            │
│ __getattr__                                                                                      │
│                                                                                                  │
│   1133 │   │   if name in self._modules:                                                         │
│   1134 │   │   │   value = self._get_module(name)                                                │
│   1135 │   │   elif name in self._class_to_module.keys():                                        │
│ ❱ 1136 │   │   │   module = self._get_module(self._class_to_module[name])                        │
│   1137 │   │   │   value = getattr(module, name)                                                 │
│   1138 │   │   else:                                                                             │
│   1139 │   │   │   raise AttributeError(f"module {self.__name__} has no attribute {name}")       │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/transformers/utils/import_utils.py:1148 in            │
│ _get_module                                                                                      │
│                                                                                                  │
│   1145 │   │   try:                                                                              │
│   1146 │   │   │   return importlib.import_module("." + module_name, self.__name__)              │
│   1147 │   │   except Exception as e:                                                            │
│ ❱ 1148 │   │   │   raise RuntimeError(                                                           │
│   1149 │   │   │   │   f"Failed to import {self.__name__}.{module_name} because of the followin  │
│   1150 │   │   │   │   f" traceback):\n{e}"                                                      │
│   1151 │   │   │   ) from e                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to import transformers.models.llama.tokenization_llama_fast because of the following error
(look up to see its traceback):
tokenizers>=0.13.3 is required for a normal functioning of this module, but found tokenizers==0.13.2.
ggml_metal_free: deallocating

[cryptoB0T]

issue resolved by updating tokenizer

pip install tokenizers==0.13.3

[sabaimran]

Glad you were able to resolve it, @cryptoB0T ! What OS was this on? And what do you get when you run khoj --version? I wasn't able to reproduce the issue.

[cryptoB0T]

OSx Ventura, Version 13.5.2 (22G91).

khoj --version 0.11.4