Closed miiPoP closed 1 year ago
Python Launcher(py
command) "prefers per-user installations over system-wide ones, and orders by language version" according to the document. Maybe you didn't use the default install option before, please uninstall all Python versions and reinstall Python 3.10 and select the default option("Install Now" button) in the Python installer.
But torch supports Python 3.8, pip should install CUDA enabled torch. Does your PC have CUDA and CUDA supported GPU?
"worddumb-libs-*" folders won't be deleted by plugin, you may want to delete the unused one to save space or let plugin reinstall dependencies. You may also need to run "py -m pip cache purge" to clear the pip cache, just in case pip has a CUDA disabled torch cache.
"please uninstall all Python versions and reinstall Python 3.10 and select the default option("Install Now" button) in the Python installer."
Done.
"Does your PC have CUDA"
Yes:
Microsoft Windows [version 10.0.19045.2311]
(c) Microsoft Corporation. Tous droits réservés.
C:\Windows\System32>nvidia-smi
Thu Dec 1 20:52:06 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 527.37 Driver Version: 527.37 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... WDDM | 00000000:09:00.0 On | N/A |
| 0% 40C P8 15W / 130W | 484MiB / 6144MiB | 2% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
...
"and CUDA supported GPU?" Yes: GeForce GTX 1660
"You may also need to run "py -m pip cache purge" to clear the pip cache, just in case pip has a CUDA disabled torch cache."
Done.
Still, though Python 3.10 is invoked now, the error persists:
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 162, in do_job
File "calibre_plugins.worddumb.utils", line 47, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', 'BB3J0DJUK4', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.27.1 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '', '-x', '-s']' returned non-zero exit status 1.
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000001B6BA9516C0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000001B6BA952800>, 'abort': <threading.Event object at 0x000001B6BA93B8E0>, 'log': <calibre.utils.logging.GUILog object at 0x000001B6BA93B820>}
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 56, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 221, in create_files
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 561, in load_spacy
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\__init__.py", line 54, in load
return util.load_model(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 432, in load_model
return load_model_from_package(name, **kwargs) # type: ignore[arg-type]
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 468, in load_model_from_package
return cls.load(vocab=vocab, disable=disable, enable=enable, exclude=exclude, config=config) # type: ignore[attr-defined]
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\en_core_web_trf\__init__.py", line 10, in load
return load_model_from_init_py(__file__, **overrides)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 649, in load_model_from_init_py
return load_model_from_path(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 506, in load_model_from_path
nlp = load_model_from_config(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 554, in load_model_from_config
nlp = lang_cls.from_config(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 1818, in from_config
nlp.add_pipe(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 801, in add_pipe
pipe_component = self.create_pipe(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 680, in create_pipe
resolved = registry.resolve(cfg, validate=validate)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 728, in resolve
resolved, _ = cls._make(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 777, in _make
filled, _, resolved = cls._fill(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 832, in _fill
filled[key], validation[v_key], final[key] = cls._fill(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 849, in _fill
getter_result = getter(*args, **kwargs)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\architectures.py", line 234, in create_TransformerModel_v3
model = TransformerModel(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\transformer_model.py", line 45, in __init__
wrapper = HFWrapper(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\hf_wrapper.py", line 53, in HFWrapper
HFShim(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\hf_shim.py", line 41, in __init__
super().__init__(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\thinc\shims\pytorch.py", line 47, in __init__
device = get_torch_default_device()
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\thinc\util.py", line 41, in get_torch_default_device
device_id = torch.cuda.current_device()
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\torch\cuda\__init__.py", line 552, in current_device
_lazy_init()
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\torch\cuda\__init__.py", line 221, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
How about running the this command py -m pip install -U -t C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10 --no-user spacy[cuda-autodetect,transformers]
manually?
The latest CUDA release is 11.8 but yours is 12.0. I find a relevant discussion at here and they said nvcc --version
is the correct command to find CUDA version.
"How about running the this command manually"
Done (without understanding much):
(c) Microsoft Corporation. Tous droits réservés.
C:\Windows\System32>py -m pip install -U -t "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10" --no-user spacy[cuda-autodetect,transformers]
Collecting spacy[cuda-autodetect,transformers]
Using cached spacy-3.4.3-cp310-cp310-win_amd64.whl (11.9 MB)
Collecting spacy-legacy<3.1.0,>=3.0.10
Using cached spacy_legacy-3.0.10-py2.py3-none-any.whl (21 kB)
Collecting numpy>=1.15.0
Using cached numpy-1.23.5-cp310-cp310-win_amd64.whl (14.6 MB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting srsly<3.0.0,>=2.4.3
Using cached srsly-2.4.5-cp310-cp310-win_amd64.whl (479 kB)
Collecting setuptools
Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting preshed<3.1.0,>=3.0.2
Using cached preshed-3.0.8-cp310-cp310-win_amd64.whl (94 kB)
Collecting typer<0.8.0,>=0.3.0
Using cached typer-0.7.0-py3-none-any.whl (38 kB)
Collecting catalogue<2.1.0,>=2.0.6
Using cached catalogue-2.0.8-py3-none-any.whl (17 kB)
Collecting cymem<2.1.0,>=2.0.2
Using cached cymem-2.0.7-cp310-cp310-win_amd64.whl (29 kB)
Collecting murmurhash<1.1.0,>=0.28.0
Using cached murmurhash-1.0.9-cp310-cp310-win_amd64.whl (18 kB)
Collecting pathy>=0.3.5
Using cached pathy-0.10.0-py3-none-any.whl (48 kB)
Collecting langcodes<4.0.0,>=3.2.0
Using cached langcodes-3.3.0-py3-none-any.whl (181 kB)
Collecting packaging>=20.0
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting tqdm<5.0.0,>=4.38.0
Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting requests<3.0.0,>=2.13.0
Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting thinc<8.2.0,>=8.1.0
Using cached thinc-8.1.5-cp310-cp310-win_amd64.whl (1.3 MB)
Collecting spacy-loggers<2.0.0,>=1.0.0
Using cached spacy_loggers-1.0.3-py3-none-any.whl (9.3 kB)
Collecting pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4
Using cached pydantic-1.10.2-cp310-cp310-win_amd64.whl (2.1 MB)
Collecting wasabi<1.1.0,>=0.9.1
Using cached wasabi-0.10.1-py3-none-any.whl (26 kB)
Collecting spacy-transformers<1.2.0,>=1.1.2
Using cached spacy_transformers-1.1.8-py2.py3-none-any.whl (53 kB)
Collecting cupy-wheel<12.0.0,>=11.0.0
Using cached cupy-wheel-11.3.0.tar.gz (3.0 kB)
Preparing metadata (setup.py) ... done
Collecting cupy-cuda11x==11.3.0
Using cached cupy_cuda11x-11.3.0-cp310-cp310-win_amd64.whl (71.9 MB)
Collecting fastrlock>=0.5
Using cached fastrlock-0.8.1-cp310-cp310-win_amd64.whl (28 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting smart-open<6.0.0,>=5.2.1
Using cached smart_open-5.2.1-py3-none-any.whl (58 kB)
Collecting typing-extensions>=4.1.0
Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting charset-normalizer<3,>=2
Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting transformers<4.22.0,>=3.4.0
Using cached transformers-4.21.3-py3-none-any.whl (4.7 MB)
Collecting torch>=1.6.0
Using cached torch-1.13.0-cp310-cp310-win_amd64.whl (167.3 MB)
Collecting spacy-alignments<1.0.0,>=0.7.2
Using cached spacy_alignments-0.8.6-cp310-cp310-win_amd64.whl (189 kB)
Collecting confection<1.0.0,>=0.0.1
Using cached confection-0.0.3-py3-none-any.whl (32 kB)
Collecting blis<0.8.0,>=0.7.8
Using cached blis-0.7.9-cp310-cp310-win_amd64.whl (7.0 MB)
Collecting colorama
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting click<9.0.0,>=7.1.1
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.1.1-cp310-cp310-win_amd64.whl (17 kB)
Collecting pyyaml>=5.1
Using cached PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
Collecting huggingface-hub<1.0,>=0.1.0
Using cached huggingface_hub-0.11.1-py3-none-any.whl (182 kB)
Collecting tokenizers!=0.11.3,<0.13,>=0.11.1
Using cached tokenizers-0.12.1-cp310-cp310-win_amd64.whl (3.3 MB)
Collecting regex!=2019.12.17
Using cached regex-2022.10.31-cp310-cp310-win_amd64.whl (267 kB)
Collecting filelock
Using cached filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: wasabi, tokenizers, fastrlock, cymem, urllib3, typing-extensions, spacy-loggers, spacy-legacy, spacy-alignments, smart-open, setuptools, regex, pyyaml, pyparsing, numpy, murmurhash, MarkupSafe, langcodes, idna, filelock, colorama, charset-normalizer, certifi, catalogue, tqdm, torch, srsly, requests, pydantic, preshed, packaging, jinja2, cupy-cuda11x, click, blis, typer, huggingface-hub, cupy-wheel, confection, transformers, thinc, pathy, spacy, spacy-transformers
DEPRECATION: cupy-wheel is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for cupy-wheel ... done
Successfully installed MarkupSafe-2.1.1 blis-0.7.9 catalogue-2.0.8 certifi-2022.9.24 charset-normalizer-2.1.1 click-8.1.3 colorama-0.4.6 confection-0.0.3 cupy-cuda11x-11.3.0 cupy-wheel-11.3.0 cymem-2.0.7 fastrlock-0.8.1 filelock-3.8.0 huggingface-hub-0.11.1 idna-3.4 jinja2-3.1.2 langcodes-3.3.0 murmurhash-1.0.9 numpy-1.23.5 packaging-21.3 pathy-0.10.0 preshed-3.0.8 pydantic-1.10.2 pyparsing-3.0.9 pyyaml-6.0 regex-2022.10.31 requests-2.28.1 setuptools-65.6.3 smart-open-5.2.1 spacy-3.4.3 spacy-alignments-0.8.6 spacy-legacy-3.0.10 spacy-loggers-1.0.3 spacy-transformers-1.1.8 srsly-2.4.5 thinc-8.1.5 tokenizers-0.12.1 torch-1.13.0 tqdm-4.64.1 transformers-4.21.3 typer-0.7.0 typing-extensions-4.4.0 urllib3-1.26.13 wasabi-0.10.1
"the correct command to find CUDA version" Right, 11.8 as you said:
C:\Windows\System32>C:\Windows\System32>nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0
Alas the problem still persists:
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 162, in do_job
File "calibre_plugins.worddumb.utils", line 47, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', 'BB3J0DJUK4', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.27.1 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '', '-x', '-s']' returned non-zero exit status 1.
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x0000025F2DAB7EB0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x0000025F2DAB7490>, 'abort': <threading.Event object at 0x0000025F36F36770>, 'log': <calibre.utils.logging.GUILog object at 0x0000025F36F37190>}
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 56, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 221, in create_files
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 561, in load_spacy
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\__init__.py", line 54, in load
return util.load_model(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 432, in load_model
return load_model_from_package(name, **kwargs) # type: ignore[arg-type]
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 468, in load_model_from_package
return cls.load(vocab=vocab, disable=disable, enable=enable, exclude=exclude, config=config) # type: ignore[attr-defined]
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\en_core_web_trf\__init__.py", line 10, in load
return load_model_from_init_py(__file__, **overrides)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 649, in load_model_from_init_py
return load_model_from_path(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 506, in load_model_from_path
nlp = load_model_from_config(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 554, in load_model_from_config
nlp = lang_cls.from_config(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 1818, in from_config
nlp.add_pipe(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 801, in add_pipe
pipe_component = self.create_pipe(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 680, in create_pipe
resolved = registry.resolve(cfg, validate=validate)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 728, in resolve
resolved, _ = cls._make(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 777, in _make
filled, _, resolved = cls._fill(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 832, in _fill
filled[key], validation[v_key], final[key] = cls._fill(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\confection\__init__.py", line 849, in _fill
getter_result = getter(*args, **kwargs)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\architectures.py", line 234, in create_TransformerModel_v3
model = TransformerModel(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\transformer_model.py", line 45, in __init__
wrapper = HFWrapper(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\hf_wrapper.py", line 53, in HFWrapper
HFShim(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\hf_shim.py", line 41, in __init__
super().__init__(
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\thinc\shims\pytorch.py", line 47, in __init__
device = get_torch_default_device()
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\thinc\util.py", line 41, in get_torch_default_device
device_id = torch.cuda.current_device()
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\torch\cuda\__init__.py", line 552, in current_device
_lazy_init()
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\torch\cuda\__init__.py", line 221, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
BTW Thanks a lot for helping but since that seems to be a CUDA installation problem on my system (more than a WordDumb one) feel free to close if you get tired.
Well... The torch package for Windows on pypi.org is the CPU build version, indeed "not compiled with CUDA enabled".
In order to install the CUDA build version, you'd have to run this command: py -m pip install -U -t C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10 --no-user torch --extra-index-url https://download.pytorch.org/whl/cu117
for CUDA 11.7. I find this on PyTorch's website: https://pytorch.org/get-started/locally. They don't list the install command for CUDA 11.8 tough, not sure if the 11.7 version would work with 11.8.
I add a commit https://github.com/xxyzz/WordDumb/commit/e7ed60d7b245c62a1c277ceb296fb264c8b3f86a to install CUDA enabled torch. But you'll have to delete torch package(rm -rf torch*
) in the worddumb-libs-py3.10 directory otherwise it won't be reinstalled.
Reporting after doing fresh install of WordDumb...
"you'd have to run this command"
Done:
Microsoft Windows [version 10.0.19045.2311]
(c) Microsoft Corporation. Tous droits réservés.
C:\Windows\System32>py -m pip install -U -t "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10" --no-user spacy[cuda-autodetect,transformers] --extra-index-url https://download.pytorch.org/whl/cu117
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu117
Collecting spacy[cuda-autodetect,transformers]
Using cached spacy-3.4.3-cp310-cp310-win_amd64.whl (11.9 MB)
Collecting tqdm<5.0.0,>=4.38.0
Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting setuptools
Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting srsly<3.0.0,>=2.4.3
Using cached srsly-2.4.5-cp310-cp310-win_amd64.whl (479 kB)
Collecting thinc<8.2.0,>=8.1.0
Using cached thinc-8.1.5-cp310-cp310-win_amd64.whl (1.3 MB)
Collecting pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4
Using cached pydantic-1.10.2-cp310-cp310-win_amd64.whl (2.1 MB)
Collecting packaging>=20.0
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting murmurhash<1.1.0,>=0.28.0
Using cached murmurhash-1.0.9-cp310-cp310-win_amd64.whl (18 kB)
Collecting typer<0.8.0,>=0.3.0
Using cached typer-0.7.0-py3-none-any.whl (38 kB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting preshed<3.1.0,>=3.0.2
Using cached preshed-3.0.8-cp310-cp310-win_amd64.whl (94 kB)
Collecting numpy>=1.15.0
Using cached numpy-1.23.5-cp310-cp310-win_amd64.whl (14.6 MB)
Collecting requests<3.0.0,>=2.13.0
Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting spacy-loggers<2.0.0,>=1.0.0
Using cached spacy_loggers-1.0.3-py3-none-any.whl (9.3 kB)
Collecting cymem<2.1.0,>=2.0.2
Using cached cymem-2.0.7-cp310-cp310-win_amd64.whl (29 kB)
Collecting wasabi<1.1.0,>=0.9.1
Using cached wasabi-0.10.1-py3-none-any.whl (26 kB)
Collecting catalogue<2.1.0,>=2.0.6
Using cached catalogue-2.0.8-py3-none-any.whl (17 kB)
Collecting spacy-legacy<3.1.0,>=3.0.10
Using cached spacy_legacy-3.0.10-py2.py3-none-any.whl (21 kB)
Collecting langcodes<4.0.0,>=3.2.0
Using cached langcodes-3.3.0-py3-none-any.whl (181 kB)
Collecting pathy>=0.3.5
Using cached pathy-0.10.0-py3-none-any.whl (48 kB)
Collecting cupy-wheel<12.0.0,>=11.0.0
Using cached cupy-wheel-11.3.0.tar.gz (3.0 kB)
Preparing metadata (setup.py) ... done
Collecting spacy-transformers<1.2.0,>=1.1.2
Using cached spacy_transformers-1.1.8-py2.py3-none-any.whl (53 kB)
Collecting cupy-cuda11x==11.3.0
Using cached cupy_cuda11x-11.3.0-cp310-cp310-win_amd64.whl (71.9 MB)
Collecting fastrlock>=0.5
Using cached fastrlock-0.8.1-cp310-cp310-win_amd64.whl (28 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting smart-open<6.0.0,>=5.2.1
Using cached smart_open-5.2.1-py3-none-any.whl (58 kB)
Collecting typing-extensions>=4.1.0
Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting charset-normalizer<3,>=2
Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting spacy-alignments<1.0.0,>=0.7.2
Using cached spacy_alignments-0.8.6-cp310-cp310-win_amd64.whl (189 kB)
Collecting torch>=1.6.0
Using cached https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp310-cp310-win_amd64.whl (2257.9 MB)
Collecting transformers<4.22.0,>=3.4.0
Using cached transformers-4.21.3-py3-none-any.whl (4.7 MB)
Collecting confection<1.0.0,>=0.0.1
Using cached confection-0.0.3-py3-none-any.whl (32 kB)
Collecting blis<0.8.0,>=0.7.8
Using cached blis-0.7.9-cp310-cp310-win_amd64.whl (7.0 MB)
Collecting colorama
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting click<9.0.0,>=7.1.1
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.1.1-cp310-cp310-win_amd64.whl (17 kB)
Collecting filelock
Using cached filelock-3.8.0-py3-none-any.whl (10 kB)
Collecting huggingface-hub<1.0,>=0.1.0
Using cached huggingface_hub-0.11.1-py3-none-any.whl (182 kB)
Collecting regex!=2019.12.17
Using cached regex-2022.10.31-cp310-cp310-win_amd64.whl (267 kB)
Collecting tokenizers!=0.11.3,<0.13,>=0.11.1
Using cached tokenizers-0.12.1-cp310-cp310-win_amd64.whl (3.3 MB)
Collecting pyyaml>=5.1
Using cached PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
Installing collected packages: wasabi, tokenizers, fastrlock, cymem, urllib3, typing-extensions, spacy-loggers, spacy-legacy, spacy-alignments, smart-open, setuptools, regex, pyyaml, pyparsing, numpy, murmurhash, MarkupSafe, langcodes, idna, filelock, colorama, charset-normalizer, certifi, catalogue, tqdm, torch, srsly, requests, pydantic, preshed, packaging, jinja2, cupy-cuda11x, click, blis, typer, huggingface-hub, cupy-wheel, confection, transformers, thinc, pathy, spacy, spacy-transformers
DEPRECATION: cupy-wheel is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for cupy-wheel ... done
Successfully installed MarkupSafe-2.1.1 blis-0.7.9 catalogue-2.0.8 certifi-2022.9.24 charset-normalizer-2.1.1 click-8.1.3 colorama-0.4.6 confection-0.0.3 cupy-cuda11x-11.3.0 cupy-wheel-11.3.0 cymem-2.0.7 fastrlock-0.8.1 filelock-3.8.0 huggingface-hub-0.11.1 idna-3.4 jinja2-3.1.2 langcodes-3.3.0 murmurhash-1.0.9 numpy-1.23.5 packaging-21.3 pathy-0.10.0 preshed-3.0.8 pydantic-1.10.2 pyparsing-3.0.9 pyyaml-6.0 regex-2022.10.31 requests-2.28.1 setuptools-65.6.3 smart-open-5.2.1 spacy-3.4.3 spacy-alignments-0.8.6 spacy-legacy-3.0.10 spacy-loggers-1.0.3 spacy-transformers-1.1.8 srsly-2.4.5 thinc-8.1.5 tokenizers-0.12.1 torch-1.13.0+cu117 tqdm-4.64.1 transformers-4.21.3 typer-0.7.0 typing-extensions-4.4.0 urllib3-1.26.13 wasabi-0.10.1
ch-1.13.0+cu117 tqdm-4.64.1 transformers-4.21.3 typer-0.7.0 typing-extensions-4.4.0 urllib3-1.26.13 wasabi-0.10.1
Create X-Ray selecting "run spaCy with GPU" now gives a new "charmap' codec can't decode" error:
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 162, in do_job
File "calibre_plugins.worddumb.utils", line 47, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', 'BB3J0DJUK4', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.27.1 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '', '-x', '-s']' returned non-zero exit status 1.
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000001DA5C95A6E0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000001DA59F8DB70>, 'abort': <threading.Event object at 0x000001DA5C9ACA60>, 'log': <calibre.utils.logging.GUILog object at 0x000001DA5C9ACB20>}
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 56, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 278, in create_files
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\database.py", line 135, in create_x_ray_db
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\utils.py", line 36, in load_json_or_pickle
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 535: character maps to <undefined>
In addition, Create X-Ray deselecting "run spaCy with GPU" is now also broken, giving a "object has no attribute 'format'" error:
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 164, in do_job
File "calibre_plugins.worddumb.parse_job", line 221, in create_files
File "calibre_plugins.worddumb.parse_job", line 561, in load_spacy
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\__init__.py", line 54, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 432, in load_model
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 468, in load_model_from_package
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\en_core_web_lg\__init__.py", line 10, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 649, in load_model_from_init_py
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 506, in load_model_from_path
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 554, in load_model_from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 1788, in from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 163, in __init__
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue\__init__.py", line 119, in get_all
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue\__init__.py", line 134, in get_entry_points
File "importlib\metadata\__init__.py", line 162, in load
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\__init__.py", line 1, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\architectures.py", line 6, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\__init__.py", line 1, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\listener.py", line 4, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\data_classes.py", line 5, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils.py", line 26, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils_base.py", line 3646, in <module>
AttributeError: 'NoneType' object has no attribute 'format'
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000210A1391270>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000210A1393340>, 'abort': <threading.Event object at 0x00000210A133B100>, 'log': <calibre.utils.logging.GUILog object at 0x00000210A133BB50>}
"not sure if the 11.7 version would work with 11.8." ...hmm, perhaps something else.
https://github.com/xxyzz/WordDumb/commit/a6f6cc58d3357d1b758fb682d69e7ce18fe6a854 should fix the UnicodeDecodeError error.
That 'NoneType' object has no attribute 'format
error is inside the "transformers" package, not sure what went wrong. This one is strange, in theory the CPU model is used, these "transformers" libraries shouldn't be called.
I add another commit: https://github.com/xxyzz/WordDumb/commit/3c797e803b45ffcf3c150e3b5a4a2caeaea20f2a, maybe this could fix the second error.
^ Updated parse_job.py, Sorry, made no difference:
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
...
[SNIP]
...
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils.py", line 26, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils_base.py", line 3646, in <module>
AttributeError: 'NoneType' object has no attribute 'format'
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000001BA988F2680>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000001BA988F2830>, 'abort': <threading.Event object at 0x000001BA98D5C070>, 'log': <calibre.utils.logging.GUILog object at 0x000001BA98D5C130>}
At least the "UnicodeDecodeError" error was fixed and X-Ray was created without error when GPU is used?
The "AttributeError" is a known error because calibre sets the optimize level to 2 and that removes all doc strings. I need to figure out why the "transformers" library is called when the "use GPU" option is not selected.
"At least the "UnicodeDecodeError" error was fixed and X-Ray was created without error when GPU is used?"
I wasn't clear: No X-Ray got created with or without using GPU.
The above was run without enabling GPU usage. I shouldn't have snipped:
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 164, in do_job
File "calibre_plugins.worddumb.parse_job", line 221, in create_files
File "calibre_plugins.worddumb.parse_job", line 562, in load_spacy
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\__init__.py", line 54, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 432, in load_model
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 468, in load_model_from_package
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\en_core_web_lg\__init__.py", line 10, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 649, in load_model_from_init_py
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 506, in load_model_from_path
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 554, in load_model_from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 1788, in from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 163, in __init__
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue\__init__.py", line 119, in get_all
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue\__init__.py", line 134, in get_entry_points
File "importlib\metadata\__init__.py", line 162, in load
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\__init__.py", line 1, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\architectures.py", line 6, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\__init__.py", line 1, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\listener.py", line 4, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\data_classes.py", line 5, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils.py", line 26, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils_base.py", line 3646, in <module>
AttributeError: 'NoneType' object has no attribute 'format'
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x0000026AFC5D0AC0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x0000026AFC5D04F0>, 'abort': <threading.Event object at 0x0000026AFC5D15A0>, 'log': <calibre.utils.logging.GUILog object at 0x0000026AFC5D19F0>}
When enabling GPU I still get the same "UnicodeDecodeError":
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 162, in do_job
File "calibre_plugins.worddumb.utils", line 47, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', 'BB3J0DJUK4', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.27.1 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '', '-x', '-s']' returned non-zero exit status 1.
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000234181B3FD0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000234169A4A00>, 'abort': <threading.Event object at 0x00000234181578B0>, 'log': <calibre.utils.logging.GUILog object at 0x0000023418157970>}
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 56, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 278, in create_files
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\database.py", line 135, in create_x_ray_db
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\utils.py", line 36, in load_json_or_pickle
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 535: character maps to <undefined>
The code line numbers in the error message indicate you're running outdated code, please install the zip file from the latest GitHub Actions Artifacts. I suspect both errors could be solved.
Mybad, instead of patching manually, I freshly reinstalled the plugin from the latest artifact worddumb-117d064104bb761e30442d83c7f15fe2fbf35993.zip
With the GPU model: now getting 'lxml' ModuleNotFoundError
calibre, version 6.9.0 (win32, embedded-python: True)
Welcome to dependency hell: Please delete the 'C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins/worddumb-libs-py*' folder then try again.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 166, in do_job
File "calibre_plugins.worddumb.utils", line 50, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', 'BB3J0DJUK4', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.27.1 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '', '-x', '-s']' returned non-zero exit status 1.
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000001CA6F51A350>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000001CA6F5194B0>, 'abort': <threading.Event object at 0x000001CA6F51B730>, 'log': <calibre.utils.logging.GUILog object at 0x000001CA6F51B8B0>}
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 56, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 303, in create_files
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\x_ray.py", line 189, in finish
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\mediawiki.py", line 322, in query_mediawiki
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\mediawiki.py", line 127, in query
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip\mediawiki.py", line 137, in get_section_text
ModuleNotFoundError: No module named 'lxml'
With the CPU model: still getting object has no attribute 'format'
calibre, version 6.9.0 (win32, embedded-python: True)
Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 168, in do_job
File "calibre_plugins.worddumb.parse_job", line 224, in create_files
File "calibre_plugins.worddumb.parse_job", line 591, in load_spacy
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\__init__.py", line 54, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 432, in load_model
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 468, in load_model_from_package
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\en_core_web_lg\__init__.py", line 10, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 649, in load_model_from_init_py
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 506, in load_model_from_path
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 554, in load_model_from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 1788, in from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 163, in __init__
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue\__init__.py", line 119, in get_all
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue\__init__.py", line 134, in get_entry_points
File "importlib\metadata\__init__.py", line 162, in load
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\__init__.py", line 1, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\architectures.py", line 6, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\__init__.py", line 1, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\layers\listener.py", line 4, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy_transformers\data_classes.py", line 5, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils.py", line 26, in <module>
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\tokenization_utils_base.py", line 3646, in <module>
AttributeError: 'NoneType' object has no attribute 'format'
Called with args: ((6291, 'AZW3', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x0000020D0B45E5F0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x0000020D0B45E680>, 'abort': <threading.Event object at 0x0000020D0B4170D0>, 'log': <calibre.utils.logging.GUILog object at 0x0000020D0B417010>}
Sorry for my incompetence with GitHub, still learning.
https://github.com/xxyzz/WordDumb/commit/3ad53d5dcb5d30c11f1704a30fe8f8922f97e326 would fix the ModuleNotFoundError
error.
Still don't have a clue about the AttributeError
error. What about delete those "spacy_transformers" and "transformers" packages before creating X-Ray with "run with GPU" disabled(this would cause error when "run with GPU" is enabled since these dependencies are missing)?
And you can use git pull
command to pull commits.
Qapla', lxml is now found and X-Ray has been created with GPU!
"What about delete those "spacy_transformers" and "transformers" packages" Sure, I will try that and report (but later).
^ OK, time to show my pip ignorance now:
Let's see what is in there:
C:\WINDOWS\system32>py -m pip list --path "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10"
Package Version
------------------ ------------
blis 0.7.9
catalogue 2.0.8
certifi 2022.9.24
charset-normalizer 2.1.1
click 8.1.3
colorama 0.4.6
confection 0.0.3
cupy-cuda11x 11.3.0
cupy-wheel 11.3.0
cymem 2.0.7
en-core-web-lg 3.4.1
en-core-web-trf 3.4.1
fastrlock 0.8.1
filelock 3.8.0
huggingface-hub 0.11.1
idna 3.4
Jinja2 3.1.2
langcodes 3.3.0
lxml 4.9.1
MarkupSafe 2.1.1
murmurhash 1.0.9
numpy 1.23.5
packaging 21.3
pathy 0.10.0
preshed 3.0.8
pydantic 1.10.2
pyparsing 3.0.9
PyYAML 6.0
rapidfuzz 2.13.2
regex 2022.10.31
requests 2.28.1
setuptools 65.6.3
smart-open 5.2.1
spacy 3.4.3
spacy-alignments 0.8.6
spacy-legacy 3.0.10
spacy-loggers 1.0.3
spacy-transformers 1.1.8
srsly 2.4.5
thinc 8.1.5
tokenizers 0.12.1
torch 1.13.0+cu117
tqdm 4.64.1
transformers 4.21.3
typer 0.7.0
typing_extensions 4.4.0
urllib3 1.26.13
wasabi 0.10.1
Why would it be as simple as:
C:\WINDOWS\system32>py -m pip uninstall spacy-transformers transformers
WARNING: Skipping spacy-transformers as it is not installed.
WARNING: Skipping transformers as it is not installed.
I just manually deleted transformers, transformers-4.21.3.dist-info, spacy_transformers, and spacy_transformers-1.1.8.dist-info directories and the X-Ray creation (via CPU) now completes successfully. No more AttributeError error.
"...this would cause error when "run with GPU" is enabled since these dependencies are missing" indeed.
En passant I had this "conversation" with this ChatGPT OpenAI bot. My questions are in bold. I am still amazed:
how do i uninstall a python package?
To uninstall a python package, you can use the pip uninstall command. For example, to uninstall the numpy package, you can use the following command:
pip uninstall numpy
This will remove the package from your system. Keep in mind that you may need to use sudo in front of the pip command if you are using a version of Python that is installed globally on your system, rather than in a virtual environment.
sudo pip uninstall numpy
You can also use the conda command if you are using the Anaconda distribution of Python. To uninstall a package using conda, you can use the conda remove command. For example, to uninstall the numpy package, you can use the following command:
conda remove numpy
This will remove the package from your system. Note that you may need to use the --force option if you are removing a package that is required by other packages on your system. For example:
conda remove numpy --force
I hope this helps! Let me know if you have any other questions.
how do I do it with pip?
To uninstall a python package using pip, you can use the pip uninstall command. For example, to uninstall the numpy package, you can use the following command:
pip uninstall numpy
This will remove the package from your system. Keep in mind that you may need to use sudo in front of the pip command if you are using a version of Python that is installed globally on your system, rather than in a virtual environment.
sudo pip uninstall numpy
You can also use the --yes or -y option to automatically answer yes to any prompts that the pip command may ask you. For example:
pip uninstall numpy --yes
Or:
pip uninstall numpy -y
I hope this helps! Let me know if you have any other questions.
why does pip list a package but says it is not installed when I run pip uninstall?
If pip lists a package but says it is not installed when you run pip uninstall, it is possible that the package is installed in a different location or in a different Python environment.
When you run pip list, it shows you the packages that are installed in the current environment. If you are using a virtual environment, then pip list will only show you the packages that are installed in that environment.
On the other hand, when you run pip uninstall, it will try to uninstall the package from the current environment. If the package is not installed in the current environment, then pip will return an error saying that the package is not installed.
To fix this, you can try activating the correct Python environment using the activate command (if you are using virtual environments) and then running pip uninstall again.
For example, if you are using the virtualenv package to manage your virtual environments, you can activate the correct environment using the following command:
source /path/to/venv/bin/activate
Replace /path/to/venv with the actual path to your virtual environment. Once the environment is activated, you can run pip uninstall to uninstall the package.
Alternatively, you can use the -P or --prefix option with pip uninstall to specify the path to the environment where the package is installed. For example:
pip uninstall numpy -P /path/to/venv
This will uninstall the numpy package from the environment located at /path/to/venv.
I hope this helps! Let me know if you have any other questions.
[EDIT: 2 hours later I added] what are python transformer and spacy-transformer packages create decode error when using non cuda code
Python transformer and spacy-transformer are two different packages that are not related to each other. The transformer package is a library for natural language processing tasks, specifically for training and evaluating transformer-based models for language understanding tasks. The spacy-transformer package is a plugin for the popular natural language processing library spaCy, which allows users to use transformer-based models trained with the transformer package in their spaCy pipelines.
As for the "decode error" you mentioned, it is not clear what you are referring to or what the error message is. However, it is worth noting that the transformer and spacy-transformer packages are designed to work with GPUs (Graphics Processing Units) for efficient and fast training and inference of transformer-based models. If you are using a non-CUDA version of Python, it is possible that you may encounter errors or performance issues when using these packages. It is recommended to use a CUDA-enabled version of Python for optimal performance with these packages.
UnicodeDecodeError: 'charmap' codec can't decode byte was the precise error
A UnicodeDecodeError with the message 'charmap' codec can't decode byte indicates that there was a problem with trying to decode some text using the character mapping (charmap) codec. This error typically occurs when there is a mismatch between the character encoding of the text and the character map being used to decode it. For example, if the text is encoded using the UTF-8 character encoding, but the character map being used to decode it is for a different encoding, such as ISO-8859-1, this error can occur.
To fix this error, you need to make sure that the character map being used to decode the text is compatible with the character encoding of the text. You can do this by specifying the correct character encoding when you load or read the text into your program. For example, if your text is encoded using UTF-8, you can specify the utf-8 encoding when you read the text using the open() function in Python, like this:
with open("my_text_file.txt", "r", encoding="utf-8") as f: text = f.read()
This will ensure that the text is properly decoded using the correct character map (in this case, UTF-8), and the UnicodeDecodeError should no longer occur.
I don't even know the pip list --path
option! What I was worried about happened, even load a non-transformer model somehow spacy-transformer
is called. Don't know how to fix that yet...
UnicodeDecodeError
happens because Python open files uses the cp1252
encoding by default on Windows but all files in this project uses utf-8
.
I don't have any CUDA supported device, but I thought add support of spaCy transformer model would just need to add a few lines of code. I was completely wrong... Thanks for testing my buggy code!
https://github.com/xxyzz/WordDumb/commit/11dd5308ad1131a6c402da7561d0e0002acbfe5d should fix AttributeError
error. I don't know why spaCy loads unused packages.
Fixed indeed. Thanks again.
Once transformers releases a new version and spacy-transformers also supports the new transformers version, I can remove the hacks to avoid the AttributeError
error in transformers which would simplify the code.
Hi @miiPoP, the latest transformers package fixed the doc string error and I have removed the monkey patch in the master branch so creating X-Ray file with GPU no longer need to run in the system Python interpreter. Could you install the test plugin from GitHub Actions to test the "Run spaCy with GPU" feature?
Sorry, it fails both using the GPU or not
`With "Run spaCy with GPU" feature, after a very long time (minutes) it fails with:
calibre, version 6.11.0 (win32, embedded-python: True) Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month
Job: "Générant X-Ray pour The Mythical Man-Month" failed with error:
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\transformers\utils\import_utils.py", line 1110, in _get_module
File "importlib__init__.py", line 126, in import_module
File "
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 180, in do_job
File "calibre_plugins.worddumb.parse_job", line 239, in create_files
File "calibre_plugins.worddumb.parse_job", line 646, in load_spacy
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy__init.py", line 54, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 432, in load_model
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 468, in load_model_from_package
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\en_core_web_trf__init__.py", line 10, in load
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 649, in load_model_from_init_py
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 506, in load_model_from_path
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\util.py", line 554, in load_model_from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 1773, in from_config
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\spacy\language.py", line 162, in init
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue__init__.py", line 119, in get_all
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\catalogue__init.py", line 134, in get_entry_points
File "importlib\metadata\init.py", line 162, in load
File "importlib\init.py", line 126, in import_module
File "
Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000002104F241FC0>, {'spacy': 'en_coreweb', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000002104D82E470>, 'abort': <threading.Event object at 0x00000210505D3040>, 'log': <calibre.utils.logging.GUILog object at 0x00000210505D2F80>}
Without "Run spaCy with GPU" feature, it almost immediately gives the error:
calibre, version 6.11.0 (win32, embedded-python: True) Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month Job: "Générant X-Ray pour The Mythical Man-Month" failed with error: Traceback (most recent call last): File "calibre\gui2\threaded_jobs.py", line 82, in start_work File "calibre_plugins.worddumb.parse_job", line 135, in do_job File "calibre_plugins.worddumb.deps", line 52, in install_deps File "calibre_plugins.worddumb.deps", line 165, in pip_install File "calibre_plugins.worddumb.utils", line 50, in run_subprocess File "subprocess.py", line 501, in run File "subprocess.py", line 966, in init File "subprocess.py", line 1375, in _execute_child File "subprocess.py", line 561, in list2cmdline File "os.py", line 822, in fsdecode TypeError: expected str, bytes or os.PathLike object, not NoneType
Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x0000021011317280>, {'spacy': 'en_coreweb', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000210113170A0>, 'abort': <threading.Event object at 0x00000210113176D0>, 'log': <calibre.utils.logging.GUILog object at 0x0000021011317790>}
`
Dang, transformers still has some code using the doc string. I restore the monkey patch in the master branch, the first error should be fixed.
But I can't make of the second error, it says the one of the args
is None
at here: https://github.com/xxyzz/WordDumb/blob/f1b0d010968aa350f57ddcc184085056324751b0/utils.py#L43-L58
I add a print
in this commit https://github.com/xxyzz/WordDumb/commit/73109c09aae0e1eb7c0b106d31333f7310e87557
please install the new test zip file and start calibre with command calibre-debug -g
then look for a line starts with "args=" in terminal.
As requested, terminal log:
` Microsoft Windows [version 10.0.19045.2546] (c) Microsoft Corporation. Tous droits réservés.
C:\Windows\System32>calibre-debug -g
calibre 6.11 embedded-python: True
Windows-10-10.0.19045-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19045')
Python 3.10.1
Windows: ('10', '10.0.19045', 'SP0', 'Multiprocessor Free')
Interface language: fr
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 50, 0) && DeDRM (10, 0, 3) && AZW6 Image Merge (1, 0, 0) && Package KFX (from KFX Input) (1, 50, 0) && APNX Generator (1, 1, 0) && AudioBook_Duration (1, 0, 9) && Count Pages (1, 12, 0) && EpubCheck (0, 2, 4) && FSRead plugin (1, 0, 0) && Find Duplicates (1, 10, 7) && Generate Cover (2, 3, 2) && Hyphenate This! (0, 1, 3) && KFX metadata reader (from KFX Input) (1, 50, 0) && KFX Input (1, 50, 0) && Set KFX metadata (from KFX Output) (1, 65, 0) && KFX Output (1, 65, 0) && KindleUnpack - The Plugin (0, 83, 1) && KindleXRay (0, 0, 4) && Obok DeDRM (10, 0, 3) && Open With (1, 8, 1) && Quality Check (1, 13, 3) && Standard Ebooks (1, 0, 0) && TTS to MP3 (0, 6, 0) && WordDumb (3, 29, 0) && X-Ray Creator (3, 2, 1)
QPA platform: windows
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 88.6690909090909 x 87.08571428571427
ApplicationPaletteChange event ignored
Using calibre Qt style: True
ApplicationPaletteChange event received
[0.00] Starting up...
[0.00] Showing splash screen...
[0.41] splash screen shown
[0.41] Initializing db...
[0.64] db initialized
[0.64] Constructing main UI...
Traceback (most recent call last):
File "calibre\gui2\ui.py", line 137, in init
File "calibre\gui2\ui.py", line 155, in init_iaction
File "calibre\customize__init.py", line 612, in load_actual_plugin
File "importlib__init.py", line 126, in import_module
File "
Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000194F8697D90>, {'spacy': 'en_coreweb', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000194F8697460>, 'abort': <threading.Event object at 0x00000194F844F760>, 'log': <calibre.utils.logging.GUILog object at 0x00000194F844F670>}
`
And Calibre log, with CUDA processing enabled:
` calibre, version 6.11.0 (win32, embedded-python: True) Tonnerre de Brest!: Une erreur s'est produite, veuillez copier le message d'erreur et rapportez le problème à GitHub.
Starting job: Générant X-Ray pour The Mythical Man-Month Job: "Générant X-Ray pour The Mythical Man-Month" failed with error: Traceback (most recent call last): File "calibre\gui2\threaded_jobs.py", line 82, in start_work File "calibre_plugins.worddumb.parse_job", line 184, in do_job File "calibre_plugins.worddumb.utils", line 51, in runsubprocess File "subprocess.py", line 524, in run subprocess.CalledProcessError: Command '['py', 'C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip', 'BB3J0DJUK4', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.29.0 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip', '', '-x', '-s']' returned non-zero exit status 1.
Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000194F8697D90>, {'spacy': 'en_coreweb', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000194F8697460>, 'abort': <threading.Event object at 0x00000194F844F760>, 'log': <calibre.utils.logging.GUILog object at 0x00000194F844F670>} C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\cuda\compiler.py:103: UserWarning: Failed to auto-detect cl.exe path: <class 'distutils.errors.DistutilsPlatformError'>: No Microsoft Visual C++ version found warnings.warn(f'Failed to auto-detect cl.exe path: {type(e)}: {e}') Traceback (most recent call last): File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\cuda\compiler.py", line 64, in _run_cc log = subprocess.check_output(cmd, cwd=cwd, env=env, File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['C:\Program', 'Files\NVIDIA', 'GPU', 'Computing', 'Toolkit\CUDA\v11.8\bin\nvcc.EXE', '-gencode=arch=compute_75,code=sm_75', '--ptx', '-DFIRSTPASS=1', '--std=c++11', '-IC:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\_core\include', '-IC:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\_core\include\cupy\_cuda\cuda-11', '-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include', '-ftz=true', 'C:\Users\NORMAN~1\AppData\Local\Temp\tmpj6gqshq\preprocess.cu']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\cuda\compiler.py", line 366, in compile_using_nvcc
_run_cc(cmd, root_dir, 'nvcc', log_stream)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\cuda\compiler.py", line 80, in _run_cc
raise NVCCException(msg)
cupy.cuda.compiler.NVCCException: nvcc
command returns non-zero exit status.
command: ['C:\Program', 'Files\NVIDIA', 'GPU', 'Computing', 'Toolkit\CUDA\v11.8\bin\nvcc.EXE', '-gencode=arch=compute_75,code=sm_75', '--ptx', '-DFIRSTPASS=1', '--std=c++11', '-IC:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\_core\include', '-IC:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\_core\include\cupy\_cuda\cuda-11', '-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include', '-ftz=true', 'C:\Users\NORMAN~1\AppData\Local\Temp\tmpj6gqshq\preprocess.cu']
return-code: 1
stdout/stderr:
nvcc fatal : Cannot find compiler 'cl.exe' in PATH
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Normand Boivin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\WordDumb.zip__main.py", line 59, in nvcc
command returns non-zero exit status.
command: ['C:\Program', 'Files\NVIDIA', 'GPU', 'Computing', 'Toolkit\CUDA\v11.8\bin\nvcc.EXE', '-gencode=arch=compute_75,code=sm_75', '--ptx', '-DFIRSTPASS=1', '--std=c++11', '-IC:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\_core\include', '-IC:\Users\Normand Boivin\AppData\Roaming\calibre\plugins\worddumb-libs-py3.10\cupy\_core\include\cupy\_cuda\cuda-11', '-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include', '-ftz=true', 'C:\Users\NORMAN~1\AppData\Local\Temp\tmpj6gqshq\preprocess.cu']
return-code: 1
stdout/stderr:
nvcc fatal : Cannot find compiler 'cl.exe' in PATH
`
The error message says cupy
tried to run nvcc
but nvcc
can't find compiler cl.exe
and print a warning: No Microsoft Visual C++ version found
. You can download the required compiler at here: https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022
And what's the output when "run with GPU" is disabled?
I also notice there are some errors caused by other plugins, but those are probably irrelevant.
My bad (Visual Studio had been de-installed in the last month), after re-installing it both WordDumb flavors work perfectly:
Without CUDA:
C:\Windows\System32>calibre-debug -g
[snip]
args=['py', '-m', 'pip', '--disable-pip-version-check', 'show', 'pip']
args=['py', '-m', 'pip', '--disable-pip-version-check', 'show', 'pip']
args=['calibre-debug', '-e', 'c:/users/normand boivin/appdata/roaming/python/python310/site-packages/pip/__pip-runner__.py', '--', '--disable-pip-version-check', 'install', '-U', '-t', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\worddumb-libs-py3.10', '--no-user', 'https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.5.0/en_core_web_lg-3.5.0-py3-none-any.whl']
Job: 0 Générant X-Ray pour The Mythical Man-Month finished
Starting job: Générant X-Ray pour The Mythical Man-Month
With CUDA:
C:\Windows\System32>calibre-debug -g
[snip]
args=['py', '-c', 'import platform; print(".".join(platform.python_version_tuple()[:2]))']
args=['py', '-m', 'pip', '--disable-pip-version-check', 'show', 'pip']
args=['py', '-c', 'import platform; print(".".join(platform.python_version_tuple()[:2]))']
args=['py', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', 'BB3J0DJUK4', 'H:\\NB\\Bibliothèque Calibre\\Frederick P. Brooks Jr_\\The Mythical Man-Month (6291)\\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', 'The_Mythical_Man-Month', '0b9790f2', 'en_core_web_trf', 'en', 'en', 'utf-8', 'WordDumb/3.29.0 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'AZW3', '1', 'C:\\Users\\Normand Boivin\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '', '-x', '-s']
Job: 0 Générant X-Ray pour The Mythical Man-Month finished
Starting job: Générant X-Ray pour The Mythical Man-Month
Checkboxes
Describe the bug
WordDumb Fail to use GPU to create X-Ray file (when CUDA is disabled everything works fine).
A fresh installation of the plugin installs worddumb-libs-py3.8, worddumb-libs-py3.10, worddumb-wikimedia in Calibre plugin directory.
But Python 3.8 is invoked instead of Python 3.10 (which I think is required, per your documentation).
If that is relevant, a separate Python 3.10 directory is in my OS PATH.
System Information
OS: Windows 10 Pro 64bit Calibre: version 6.9.0 64bit Python: version 3.10.8 64bit Plugin: version 3.27.1 (+"space in user name" patch)
Error message
Reproduce steps
Screenshots or videos
No response