search

xxyzz / WordDumb

A calibre plugin that generates Kindle Word Wise and X-Ray files for KFX, AZW3, MOBI and EPUB eBook.
https://xxyzz.github.io/WordDumb/
GNU General Public License v3.0
386 stars 19 forks source link

Error loading `cufftw64_11.dll` #162

Closed roland-pep closed 6 months ago

roland-pep commented 1 year ago

Checkboxes

Describe the bug

Using spacy with GPU. Spacy model size: Large. Cuda version installed: 12.3

The key error message is:

OSError: [WinError 127] The specified procedure could not be found. Error loading "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\torch\lib\cufftw64_11.dll" or one of its dependencies.

The cufftw64_11.dll file is in the specified folder.

Operating System name and version

Windows 11 22H2

Python version

3.12

calibre version

6.19.1

WordDumb plugin version

3.29.7

Error message

calibre, version 6.19.1 (win32, embedded-python: True)
Tonnerre de Brest!: An error occurred, please copy error message then report bug at GitHub.

Starting job: Generating Word Wise and X-Ray for Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones 
Job: "Generating Word Wise and X-Ray for Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones" failed with error: 
Traceback (most recent call last):
  File "calibre\gui2\threaded_jobs.py", line 82, in start_work
  File "calibre_plugins.worddumb.parse_job", line 203, in do_job
  File "calibre_plugins.worddumb.utils", line 55, in run_subprocess
  File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\rolan\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '{"book_id": 179, "book_path": "C:\\\\Users\\\\rolan\\\\OneDrive\\\\ebooks\\\\James Clear\\\\Atomic Habits_ An Easy & Proven Way (179)\\\\Atomic Habits_ An Easy & Proven - James Clear.azw3", "mi": null, "book_fmt": "AZW3", "book_lang": "en", "useragent": "WordDumb/3.29.7 (https://github.com/xxyzz/WordDumb)", "plugin_path": "C:\\\\Users\\\\rolan\\\\AppData\\\\Roaming\\\\calibre\\\\plugins\\\\WordDumb.zip", "spacy_model": "en_core_web_trf", "create_ww": true, "create_x": true, "asin": "B01N5AX61W", "acr": "CR!HMYWJZZXBX6HH45NC5AYSCXJFAXE", "revision": "9804cf57", "update_asin": false, "kfx_json": null, "mobi_html": null, "mobi_codec": "utf-8"}', '{"use_pos": true, "search_people": true, "model_size": "lg", "zh_wiki_variant": "cn", "fandom": "", "add_locator_map": false, "preferred_formats": ["KFX", "AZW3", "AZW", "MOBI", "EPUB"], "use_all_formats": false, "minimal_x_ray_count": 1, "en_ipa": "ga_ipa", "zh_ipa": "pinyin", "choose_format_manually": true, "wiktionary_gloss_lang": "en", "kindle_gloss_lang": "en", "use_gpu": true, "cuda": "cu121", "last_opened_kindle_lemmas_language": "ca", "last_opened_wiktionary_lemmas_language": "ca", "use_wiktionary_for_kindle": false, "ca_wiktionary_difficulty_limit": 5, "da_wiktionary_difficulty_limit": 5, "de_wiktionary_difficulty_limit": 5, "el_wiktionary_difficulty_limit": 5, "en_wiktionary_difficulty_limit": 5, "es_wiktionary_difficulty_limit": 5, "fi_wiktionary_difficulty_limit": 5, "fr_wiktionary_difficulty_limit": 5, "hr_wiktionary_difficulty_limit": 5, "it_wiktionary_difficulty_limit": 5, "ja_wiktionary_difficulty_limit": 5, "ko_wiktionary_difficulty_limit": 5, "lt_wiktionary_difficulty_limit": 5, "mk_wiktionary_difficulty_limit": 5, "nl_wiktionary_difficulty_limit": 5, "no_wiktionary_difficulty_limit": 5, "pl_wiktionary_difficulty_limit": 5, "pt_wiktionary_difficulty_limit": 5, "ro_wiktionary_difficulty_limit": 5, "ru_wiktionary_difficulty_limit": 5, "sl_wiktionary_difficulty_limit": 5, "sv_wiktionary_difficulty_limit": 5, "uk_wiktionary_difficulty_limit": 5, "zh_wiktionary_difficulty_limit": 5}']' returned non-zero exit status 1.

Called with args: (ParseJobData(book_id=179, book_path='C:\\Users\\rolan\\OneDrive\\ebooks\\James Clear\\Atomic Habits_ An Easy & Proven Way (179)\\Atomic Habits_ An Easy & Proven - James Clear.azw3', mi=<calibre.ebooks.metadata.book.base.Metadata object at 0x000001D0C24B1360>, book_fmt='AZW3', book_lang='en', useragent='WordDumb/3.29.7 (https://github.com/xxyzz/WordDumb)', plugin_path='C:\\Users\\rolan\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', spacy_model='en_core_web_trf', create_ww=True, create_x=True, asin='B01N5AX61W', acr='CR!HMYWJZZXBX6HH45NC5AYSCXJFAXE', revision='9804cf57', update_asin=False, kfx_json=None, mobi_html=None, mobi_codec='utf-8'),) {'notifications': <queue.Queue object at 0x000001D0C24B18D0>, 'abort': <threading.Event object at 0x000001D0C24B0FD0>, 'log': <calibre.utils.logging.GUILog object at 0x000001D0C24B16F0>} 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 39, in <module>
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 229, in create_files
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 771, in load_spacy
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\__init__.py", line 6, in <module>
    from .errors import setup_default_warnings
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\errors.py", line 3, in <module>
    from .compat import Literal
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\compat.py", line 4, in <module>
    from thinc.util import copy_array
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\thinc\__init__.py", line 5, in <module>
    from .config import registry
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\thinc\config.py", line 5, in <module>
    from .types import Decorator
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\thinc\types.py", line 25, in <module>
    from .compat import cupy, has_cupy
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\thinc\compat.py", line 32, in <module>
    import torch
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\torch\__init__.py", line 128, in <module>
    raise err
OSError: [WinError 127] The specified procedure could not be found. Error loading "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\torch\lib\cufftw64_11.dll" or one of its dependencies.

Reproduce steps

  1. Tick Run spaCy with GPU
  2. Set spaCy model size to Large.
  3. Click Create Word Wise and X-Ray

Generated files, screenshots or videos

No response

xxyzz commented 1 year ago

Does another folder in the "worddumb-libs-py3.11" also has name starts with "torch"?

roland-pep commented 1 year ago

Does another folder in the "worddumb-libs-py3.11" also has name starts with "torch"?

Yes, it does. The following folders starts with the name torch: _

torch torch-2.1.0.dist-info torch-2.1.0+cu121.dist-info torchgen

_

xxyzz commented 1 year ago

The CPU package(torch folder) should be deleted, https://github.com/xxyzz/WordDumb/commit/57f975b9b05722306d3d6ffaae00cdfd2410fd0b should fix this error.

roland-pep commented 1 year ago

If I delete torch folder, I get the following error:

calibre, version 6.29.0 (win32, embedded-python: True)
Welcome to dependency hell: Please delete the 'C:\Users\rolan\AppData\Roaming\calibre\plugins/worddumb-libs-py*' folder then try again.

Starting job: Generating Word Wise and X-Ray for Tao of React: Beginner to Intermediate Guide to Mastering Design Patterns — 80+ Principles for Building Extensible, Testable, and Maintainable React.js Applications 
Job: "Generating Word Wise and X-Ray for Tao of React: Beginner to Intermediate Guide to Mastering Design Patterns — 80+ Principles for Building Extensible, Testable, and Maintainable React.js Applications" failed with error: 
Traceback (most recent call last):
  File "calibre\gui2\threaded_jobs.py", line 82, in start_work
  File "calibre_plugins.worddumb.parse_job", line 203, in do_job
  File "calibre_plugins.worddumb.utils", line 55, in run_subprocess
  File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\rolan\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', '{"book_id": 198, "book_path": "C:\\\\Users\\\\rolan\\\\OneDrive\\\\ebooks\\\\Alexander Kondov\\\\Tao of React_ Beginner to Intermedi (198)\\\\Tao of React_ Beginner to Inter - Alexander Kondov.kfx", "mi": null, "book_fmt": "KFX", "book_lang": "en", "useragent": "WordDumb/3.29.8 (https://github.com/xxyzz/WordDumb)", "plugin_path": "C:\\\\Users\\\\rolan\\\\AppData\\\\Roaming\\\\calibre\\\\plugins\\\\WordDumb.zip", "spacy_model": "en_core_web_trf", "create_ww": true, "create_x": true, "asin": "BBSN7ZXEG2", "acr": "CR!B295NC6502FFX9FSV1DZM08R8QZ8", "revision": "", "update_asin": false, "kfx_json": null, "mobi_html": null, "mobi_codec": ""}', '{"use_pos": true, "search_people": true, "model_size": "lg", "zh_wiki_variant": "cn", "fandom": "", "add_locator_map": false, "preferred_formats": ["KFX", "AZW3", "AZW", "MOBI", "EPUB"], "use_all_formats": false, "minimal_x_ray_count": 1, "en_ipa": "ga_ipa", "zh_ipa": "pinyin", "choose_format_manually": true, "wiktionary_gloss_lang": "en", "kindle_gloss_lang": "en", "use_gpu": true, "cuda": "cu121", "last_opened_kindle_lemmas_language": "ca", "last_opened_wiktionary_lemmas_language": "ca", "use_wiktionary_for_kindle": false, "ca_wiktionary_difficulty_limit": 5, "da_wiktionary_difficulty_limit": 5, "de_wiktionary_difficulty_limit": 5, "el_wiktionary_difficulty_limit": 5, "en_wiktionary_difficulty_limit": 5, "es_wiktionary_difficulty_limit": 5, "fi_wiktionary_difficulty_limit": 5, "fr_wiktionary_difficulty_limit": 5, "hr_wiktionary_difficulty_limit": 5, "it_wiktionary_difficulty_limit": 5, "ja_wiktionary_difficulty_limit": 5, "ko_wiktionary_difficulty_limit": 5, "lt_wiktionary_difficulty_limit": 5, "mk_wiktionary_difficulty_limit": 5, "nl_wiktionary_difficulty_limit": 5, "no_wiktionary_difficulty_limit": 5, "pl_wiktionary_difficulty_limit": 5, "pt_wiktionary_difficulty_limit": 5, "ro_wiktionary_difficulty_limit": 5, "ru_wiktionary_difficulty_limit": 5, "sl_wiktionary_difficulty_limit": 5, "sv_wiktionary_difficulty_limit": 5, "uk_wiktionary_difficulty_limit": 5, "zh_wiktionary_difficulty_limit": 5}']' returned non-zero exit status 1.

Called with args: (ParseJobData(book_id=198, book_path='C:\\Users\\rolan\\OneDrive\\ebooks\\Alexander Kondov\\Tao of React_ Beginner to Intermedi (198)\\Tao of React_ Beginner to Inter - Alexander Kondov.kfx', mi=<calibre.ebooks.metadata.book.base.Metadata object at 0x00000179CD7291B0>, book_fmt='KFX', book_lang='en', useragent='WordDumb/3.29.8 (https://github.com/xxyzz/WordDumb)', plugin_path='C:\\Users\\rolan\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip', spacy_model='en_core_web_trf', create_ww=True, create_x=True, asin='BBSN7ZXEG2', acr='CR!B295NC6502FFX9FSV1DZM08R8QZ8', revision='', update_asin=False, kfx_json=None, mobi_html=None, mobi_codec=''),) {'notifications': <queue.Queue object at 0x00000179CD7287C0>, 'abort': <threading.Event object at 0x00000179CD729570>, 'log': <calibre.utils.logging.GUILog object at 0x00000179CD7293F0>} 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\WordDumb.zip\__main__.py", line 39, in <module>
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 229, in create_files
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 786, in load_spacy
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\__init__.py", line 51, in load
    return util.load_model(
           ^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\util.py", line 465, in load_model
    return load_model_from_package(name, **kwargs)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\util.py", line 501, in load_model_from_package
    return cls.load(vocab=vocab, disable=disable, enable=enable, exclude=exclude, config=config)  # type: ignore[attr-defined]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\en_core_web_trf\__init__.py", line 10, in load
    return load_model_from_init_py(__file__, **overrides)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\util.py", line 682, in load_model_from_init_py
    return load_model_from_path(
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\util.py", line 539, in load_model_from_path
    nlp = load_model_from_config(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\util.py", line 587, in load_model_from_config
    nlp = lang_cls.from_config(
          ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\language.py", line 1830, in from_config
    nlp = lang_cls(
          ^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy\language.py", line 188, in __init__
    util.registry._entry_point_factories.get_all()
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\catalogue\__init__.py", line 110, in get_all
    result.update(self.get_entry_points())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\catalogue\__init__.py", line 125, in get_entry_points
    result[entry_point.name] = entry_point.load()
                               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Local\Programs\Python\Python311\Lib\importlib\metadata\__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rolan\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy_curated_transformers\pipeline\__init__.py", line 1, in <module>
    from .transformer import CuratedTransformer
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy_curated_transformers\pipeline\transformer.py", line 26, in <module>
    from ..models.listeners import ListenerStateUtils
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy_curated_transformers\models\__init__.py", line 1, in <module>
    from .architectures import (
  File "C:\Users\rolan\AppData\Roaming\calibre\plugins\worddumb-libs-py3.11\spacy_curated_transformers\models\architectures.py", line 5, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
xxyzz commented 1 year ago

Sorry I though the GPU torch package is not in the torch folder, this folder shouldn't be deleted. You can delete the torch-2.1.0+cu121.dist-info and the plugin will re-download the torch package.

pytorch 2.1.0 only support CUDA 12.1, you should downgrade to that version.

benjivm commented 8 months ago

Thanks @xxyzz I was able to fix this by downgrading CUDA to 12.1. Might not be exactly the same issue @roland-pep was having but I was receiving the same error.