Open jpferraro1 opened 2 months ago
Hey are you sure you have tokenizers
installed? pip install tokenizers
Hi, I also have this problem.
Transformer: 4.41.2
CUDA: 12.4
Pytorch: 2.4.0
Python : 3.11.9
GPU: RTX 4070
When I try:
from transformers import RobertaTokenizer
I get the following error:
ModuleNotFoundError Traceback (most recent call last)
File d:\Apps\envs\cuda11\Lib\site-packages\transformers\utils\import_utils.py:1535, in _LazyModule._get_module(self, module_name)
1534 try:
-> 1535 return importlib.import_module("." + module_name, self.__name__)
1536 except Exception as e:
File d:\Apps\envs\cuda11\Lib\importlib\__init__.py:126, in import_module(name, package)
125 level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)
File <frozen importlib._bootstrap>:1204, in _gcd_import(name, package, level)
File <frozen importlib._bootstrap>:1176, in _find_and_load(name, import_)
File <frozen importlib._bootstrap>:1126, in _find_and_load_unlocked(name, import_)
File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)
File <frozen importlib._bootstrap>:1204, in _gcd_import(name, package, level)
File <frozen importlib._bootstrap>:1176, in _find_and_load(name, import_)
File <frozen importlib._bootstrap>:1147, in _find_and_load_unlocked(name, import_)
...
1539 f" traceback):\n{e}"
1540 ) from e
RuntimeError: Failed to import transformers.models.roberta because of the following error (look up to see its traceback):
No module named 'tokenizers.metrics'
It looks like Transformers 4.44.2 installs tokenizers 0.19.0 which does not work with Python 3.9.19. When I loaded tokenizers 0.19.1 it worked. Tokenizers 0.20 is not compatible.
From: Mohamed JALANJI @.> Reply-To: huggingface/tokenizers @.> Date: Monday, August 26, 2024 at 2:20 PM To: huggingface/tokenizers @.> Cc: Jeffrey Page Ferraro @.>, Author @.***> Subject: Re: [huggingface/tokenizers] ModuleNotFoundError: No module named 'tokenizers.tokenizers' (Issue #1619)
Hi, I also have this problem. When I try: from transformers import RobertaTokenizer
I get the following error:
ModuleNotFoundError Traceback (most recent call last)
File d:\Apps\envs\cuda11\Lib\site-packages\transformers\utils\import_utils.py:1535, in _LazyModule._get_module(self, module_name)
1534 try:
-> 1535 return importlib.import_module("." + module_name, self.name)
1536 except Exception as e:
File d:\Apps\envs\cuda11\Lib\importlib__init__.py:126, in import_module(name, package)
125 level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)
File
File
File
File
File
File
File
...
1539 f" traceback):\n{e}"
1540 ) from e
RuntimeError: Failed to import transformers.models.roberta because of the following error (look up to see its traceback):
No module named 'tokenizers.metrics'
— Reply to this email directly, view it on GitHubhttps://github.com/huggingface/tokenizers/issues/1619#issuecomment-2311006696, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A76SFUD42AU4WWF5D7E56IDZTOEIPAVCNFSM6AAAAABNC3GKCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGAYDMNRZGY. You are receiving this because you authored the thread.Message ID: @.***>
It looks like Transformers 4.44.2 installs tokenizers 0.19.0 which does not work with Python 3.9.19. When I loaded tokenizers 0.19.1 it worked. Tokenizers 0.20 is not compatible.
But when I install Transformers I get the version 4.41.2
, should I install the 4.44.2
?
@mohjalanji yes
Is Transformers 4.44.0 compatible with Python 3.9.19? I receive the following error below - any thoughts are appreciated.
import transformers from transformers import AutoTokenizer
Produces the following error:
ModuleNotFoundError Traceback (most recent call last) D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1602 try: -> 1603 return importlib.import_module("." + module_name, self.name) 1604 except Exception as e:
D:\Anaconda3\lib\importlib__init__.py in import_module(name, package) 126 level += 1 --> 127 return _bootstrap._gcd_import(name[level:], package, level) 128
D:\Anaconda3\lib\importlib_bootstrap.py in _gcd_import(name, package, level)
D:\Anaconda3\lib\importlib_bootstrap.py in _find_andload(name, import)
D:\Anaconda3\lib\importlib_bootstrap.py in _find_and_loadunlocked(name, import)
D:\Anaconda3\lib\importlib_bootstrap.py in _load_unlocked(spec)
D:\Anaconda3\lib\importlib_bootstrap_external.py in exec_module(self, module)
D:\Anaconda3\lib\importlib_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)
D:\Anaconda3\lib\site-packages\transformers\integrations\ggml.py in
23 import numpy as np
---> 24 from tokenizers import Tokenizer, decoders, normalizers, pre_tokenizers
25 from tokenizers.models import BPE
D:\Anaconda3\lib\site-packages\tokenizers__init__.py in
77
---> 78 from .tokenizers import (
79 AddedToken,
ModuleNotFoundError: No module named 'tokenizers.tokenizers'
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last) D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1602 try: -> 1603 return importlib.import_module("." + module_name, self.name) 1604 except Exception as e:
D:\Anaconda3\lib\importlib__init__.py in import_module(name, package) 126 level += 1 --> 127 return _bootstrap._gcd_import(name[level:], package, level) 128
D:\Anaconda3\lib\importlib_bootstrap.py in _gcd_import(name, package, level)
D:\Anaconda3\lib\importlib_bootstrap.py in _find_andload(name, import)
D:\Anaconda3\lib\importlib_bootstrap.py in _find_and_loadunlocked(name, import)
D:\Anaconda3\lib\importlib_bootstrap.py in _load_unlocked(spec)
D:\Anaconda3\lib\importlib_bootstrap_external.py in exec_module(self, module)
D:\Anaconda3\lib\importlib_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)
D:\Anaconda3\lib\site-packages\transformers\models\auto\tokenization_auto.py in
23
---> 24 from ...configuration_utils import PretrainedConfig
25 from ...dynamic_module_utils import get_class_from_dynamic_module, resolve_trust_remote_code
D:\Anaconda3\lib\site-packages\transformers\configuration_utils.py in
28 from .dynamic_module_utils import custom_object_save
---> 29 from .modeling_gguf_pytorch_utils import load_gguf_checkpoint
30 from .utils import (
D:\Anaconda3\lib\site-packages\transformers\modeling_gguf_pytorch_utils.py in
21
---> 22 from .integrations import (
23 GGUF_CONFIG_MAPPING,
D:\Anaconda3\lib\importlib_bootstrap.py in _handlefromlist(module, fromlist, import, recursive)
D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in getattr(self, name) 1592 elif name in self._class_to_module.keys(): -> 1593 module = self._get_module(self._class_to_module[name]) 1594 value = getattr(module, name)
D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1604 except Exception as e: -> 1605 raise RuntimeError( 1606 f"Failed to import {self.name}.{module_name} because of the following error (look up to see its"
RuntimeError: Failed to import transformers.integrations.ggml because of the following error (look up to see its traceback): No module named 'tokenizers.tokenizers'
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last) ~\AppData\Local\Temp\ipykernel_4232\77889138.py in
----> 1 from transformers import AutoTokenizer
D:\Anaconda3\lib\importlib_bootstrap.py in _handlefromlist(module, fromlist, import, recursive)
D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in getattr(self, name) 1592 elif name in self._class_to_module.keys(): 1593 module = self._get_module(self._class_to_module[name]) -> 1594 value = getattr(module, name) 1595 else: 1596 raise AttributeError(f"module {self.name} has no attribute {name}")
D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in getattr(self, name) 1591 value = self._get_module(name) 1592 elif name in self._class_to_module.keys(): -> 1593 module = self._get_module(self._class_to_module[name]) 1594 value = getattr(module, name) 1595 else:
D:\Anaconda3\lib\site-packages\transformers\utils\import_utils.py in _get_module(self, module_name) 1603 return importlib.import_module("." + module_name, self.name) 1604 except Exception as e: -> 1605 raise RuntimeError( 1606 f"Failed to import {self.name}.{module_name} because of the following error (look up to see its" 1607 f" traceback):\n{e}"
RuntimeError: Failed to import transformers.models.auto.tokenization_auto because of the following error (look up to see its traceback): Failed to import transformers.integrations.ggml because of the following error (look up to see its traceback): No module named 'tokenizers.tokenizers'