ModuleNotFoundError: No module named 'tokenizers.tokenizers'

[jpferraro1]

Is Transformers 4.44.0 compatible with Python 3.9.19? I receive the following error below - any thoughts are appreciated.

import transformers from transformers import AutoTokenizer

Produces the following error:

---

ModuleNotFoundError Traceback (most recent call last) D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1602 try: -> 1603 return importlib.import_module("." + module_name, self.name) 1604 except Exception as e:

D:\Anaconda3\lib\importlib__init__.py in import_module(name, package) 126 level += 1 --> 127 return _bootstrap._gcd_import(name[level:], package, level) 128

D:\Anaconda3\lib\importlib_bootstrap.py in _gcd_import(name, package, level)

D:\Anaconda3\lib\importlib_bootstrap.py in _find_andload(name, import)

D:\Anaconda3\lib\importlib_bootstrap.py in _find_and_loadunlocked(name, import)

D:\Anaconda3\lib\importlib_bootstrap.py in _load_unlocked(spec)

D:\Anaconda3\lib\importlib_bootstrap_external.py in exec_module(self, module)

D:\Anaconda3\lib\importlib_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

D:\Anaconda3\lib\site-packages\transformers\integrations\ggml.py in 23 import numpy as np ---> 24 from tokenizers import Tokenizer, decoders, normalizers, pre_tokenizers 25 from tokenizers.models import BPE

D:\Anaconda3\lib\site-packages\tokenizers__init__.py in 77 ---> 78 from .tokenizers import ( 79 AddedToken,

ModuleNotFoundError: No module named 'tokenizers.tokenizers'

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last) D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1602 try: -> 1603 return importlib.import_module("." + module_name, self.name) 1604 except Exception as e:

D:\Anaconda3\lib\importlib__init__.py in import_module(name, package) 126 level += 1 --> 127 return _bootstrap._gcd_import(name[level:], package, level) 128

D:\Anaconda3\lib\importlib_bootstrap.py in _gcd_import(name, package, level)

D:\Anaconda3\lib\importlib_bootstrap.py in _find_andload(name, import)

D:\Anaconda3\lib\importlib_bootstrap.py in _find_and_loadunlocked(name, import)

D:\Anaconda3\lib\importlib_bootstrap.py in _load_unlocked(spec)

D:\Anaconda3\lib\importlib_bootstrap_external.py in exec_module(self, module)

D:\Anaconda3\lib\importlib_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

D:\Anaconda3\lib\site-packages\transformers\models\auto\tokenization_auto.py in 23 ---> 24 from ...configuration_utils import PretrainedConfig 25 from ...dynamic_module_utils import get_class_from_dynamic_module, resolve_trust_remote_code

D:\Anaconda3\lib\site-packages\transformers\configuration_utils.py in 28 from .dynamic_module_utils import custom_object_save ---> 29 from .modeling_gguf_pytorch_utils import load_gguf_checkpoint 30 from .utils import (

D:\Anaconda3\lib\site-packages\transformers\modeling_gguf_pytorch_utils.py in 21 ---> 22 from .integrations import ( 23 GGUF_CONFIG_MAPPING,

D:\Anaconda3\lib\importlib_bootstrap.py in _handlefromlist(module, fromlist, import, recursive)

D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in getattr(self, name) 1592 elif name in self._class_to_module.keys(): -> 1593 module = self._get_module(self._class_to_module[name]) 1594 value = getattr(module, name)

D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1604 except Exception as e: -> 1605 raise RuntimeError( 1606 f"Failed to import {self.name}.{module_name} because of the following error (look up to see its"

RuntimeError: Failed to import transformers.integrations.ggml because of the following error (look up to see its traceback): No module named 'tokenizers.tokenizers'

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last) ~\AppData\Local\Temp\ipykernel_4232\77889138.py in ----> 1 from transformers import AutoTokenizer

D:\Anaconda3\lib\importlib_bootstrap.py in _handlefromlist(module, fromlist, import, recursive)

D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in getattr(self, name) 1592 elif name in self._class_to_module.keys(): 1593 module = self._get_module(self._class_to_module[name]) -> 1594 value = getattr(module, name) 1595 else: 1596 raise AttributeError(f"module {self.name} has no attribute {name}")

D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in getattr(self, name) 1591 value = self._get_module(name) 1592 elif name in self._class_to_module.keys(): -> 1593 module = self._get_module(self._class_to_module[name]) 1594 value = getattr(module, name) 1595 else:

D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1603 return importlib.import_module("." + module_name, self.name) 1604 except Exception as e: -> 1605 raise RuntimeError( 1606 f"Failed to import {self.name}.{module_name} because of the following error (look up to see its" 1607 f" traceback):\n{e}"

RuntimeError: Failed to import transformers.models.auto.tokenization_auto because of the following error (look up to see its traceback): Failed to import transformers.integrations.ggml because of the following error (look up to see its traceback): No module named 'tokenizers.tokenizers'

[ArthurZucker]

Hey are you sure you have tokenizers installed? pip install tokenizers

[mohjalanji]

Hi, I also have this problem. Transformer: 4.41.2 CUDA: 12.4 Pytorch: 2.4.0 Python : 3.11.9 GPU: RTX 4070

When I try: from transformers import RobertaTokenizer

I get the following error:

ModuleNotFoundError                       Traceback (most recent call last)
File d:\Apps\envs\cuda11\Lib\site-packages\transformers\utils\import_utils.py:1535, in _LazyModule._get_module(self, module_name)
   1534 try:
-> 1535     return importlib.import_module("." + module_name, self.__name__)
   1536 except Exception as e:

File d:\Apps\envs\cuda11\Lib\importlib\__init__.py:126, in import_module(name, package)
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1204, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1176, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1126, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File <frozen importlib._bootstrap>:1204, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1176, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1147, in _find_and_load_unlocked(name, import_)
...
   1539         f" traceback):\n{e}"
   1540     ) from e

RuntimeError: Failed to import transformers.models.roberta because of the following error (look up to see its traceback):
No module named 'tokenizers.metrics'

[jpferraro1]

It looks like Transformers 4.44.2 installs tokenizers 0.19.0 which does not work with Python 3.9.19. When I loaded tokenizers 0.19.1 it worked. Tokenizers 0.20 is not compatible.

From: Mohamed JALANJI @.> Reply-To: huggingface/tokenizers @.> Date: Monday, August 26, 2024 at 2:20 PM To: huggingface/tokenizers @.> Cc: Jeffrey Page Ferraro @.>, Author @.***> Subject: Re: [huggingface/tokenizers] ModuleNotFoundError: No module named 'tokenizers.tokenizers' (Issue #1619)

Hi, I also have this problem. When I try: from transformers import RobertaTokenizer

I get the following error:

ModuleNotFoundError Traceback (most recent call last)

File d:\Apps\envs\cuda11\Lib\site-packages\transformers\utils\import_utils.py:1535, in _LazyModule._get_module(self, module_name)

1534 try:

-> 1535 return importlib.import_module("." + module_name, self.name)

1536 except Exception as e:

File d:\Apps\envs\cuda11\Lib\importlib__init__.py:126, in import_module(name, package)

125         level += 1

--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File :1204, in _gcd_import(name, package, level)

File :1176, in _find_andload(name, import)

File :1126, in _find_and_loadunlocked(name, import)

File :241, in _call_with_frames_removed(f, *args, **kwds)

File :1204, in _gcd_import(name, package, level)

File :1176, in _find_andload(name, import)

File :1147, in _find_and_loadunlocked(name, import)

...

1539 f" traceback):\n{e}"

1540 ) from e

RuntimeError: Failed to import transformers.models.roberta because of the following error (look up to see its traceback):

No module named 'tokenizers.metrics'

— Reply to this email directly, view it on GitHubhttps://github.com/huggingface/tokenizers/issues/1619#issuecomment-2311006696, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A76SFUD42AU4WWF5D7E56IDZTOEIPAVCNFSM6AAAAABNC3GKCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGAYDMNRZGY. You are receiving this because you authored the thread.Message ID: @.***>

[jpferraro1]

It looks like Transformers 4.44.2 installs tokenizers 0.19.0 which does not work with Python 3.9.19. When I loaded tokenizers 0.19.1 it worked. Tokenizers 0.20 is not compatible.

[mohjalanji]

But when I install Transformers I get the version 4.41.2, should I install the 4.44.2?

[ArthurZucker]

@mohjalanji yes