xxyzz / WordDumb

A calibre plugin that generates Kindle Word Wise and X-Ray files for KFX, AZW3, MOBI and EPUB eBook.
https://xxyzz.github.io/WordDumb/
GNU General Public License v3.0
386 stars 19 forks source link

Not able to create file with CUDA because of torch 2.3 #213

Closed py930828 closed 3 months ago

py930828 commented 6 months ago

Checkboxes

Describe the bug and copy the error message

I'm on windows 11, CUDA version 12.1, Worddumb 3.31.4

The error is torch can't find its dependencies. I searched online that this error might be caused by torch 2.3 and people can fix it by downgrade to 2.2, however I'm not sure how I can do it in worddumb-libs-py3.12 as I don't see virtual environment setup there.

Plugin settings and reproduce steps

{ "add_locator_map": false, "choose_format_manually": true, "cuda": "cu121", "fandom": "", "kindle_gloss_lang": "zh_cn", "minimal_x_ray_count": 1, "model_size": "lg", "preferred_formats": [ "AZW3", "MOBI", "AZW", "EPUB", "KFX" ], "python_path": "", "remove_link_styles": false, "search_people": true, "use_all_formats": false, "use_gpu": true, "use_pos": true, "use_wiktionary_for_kindle": true, "wiktionary_gloss_lang": "zh_cn", "zh_wiki_variant": "cn" }

Generated files, screenshots or videos

Starting job: Generating Word Wise and X-Ray for A Case for Christ Job: "Generating Word Wise and X-Ray for A Case for Christ" failed with error: Traceback (most recent call last): File "calibre\gui2\threaded_jobs.py", line 85, in start_work File "calibre_plugins.worddumb.parse_job", line 201, in do_job File "calibre_plugins.worddumb.utils", line 52, in run_subprocess File "subprocess.py", line 571, in run subprocess.CalledProcessError: Command '['python', 'C:\Users\cl930\AppData\Roaming\calibre\plugins\WordDumb.zip', '{"book_id": 148, "book_path": "F:\\Calibre\\Calibre Library\\Steve Hays\\A Case for Christ (148)\\A Case for Christ - Steve Hays.azw3", "mi": null, "book_fmt": "AZW3", "book_lang": "en", "useragent": "WordDumb/3.31.4 (https://github.com/xxyzz/WordDumb)", "plugin_path": "C:\\Users\\cl930\\AppData\\Roaming\\calibre\\plugins\\WordDumb.zip", "spacy_model": "en_core_web_trf", "create_ww": true, "create_x": true, "asin": "BBKMF8ZBLG", "acr": "A_Case_for_Christ", "revision": "07efc666", "kfx_json": null, "mobi_html": null, "mobi_codec": "utf-8"}', '{"use_pos": true, "search_people": true, "model_size": "lg", "zh_wiki_variant": "cn", "fandom": "", "add_locator_map": false, "preferred_formats": ["AZW3", "MOBI", "AZW", "EPUB", "KFX"], "use_all_formats": false, "minimal_x_ray_count": 1, "choose_format_manually": true, "wiktionary_gloss_lang": "zh_cn", "kindle_gloss_lang": "zh_cn", "use_gpu": true, "cuda": "cu121", "use_wiktionary_for_kindle": true, "remove_link_styles": false, "python_path": "", "ca_wiktionary_difficulty_limit": 5, "cs_wiktionary_difficulty_limit": 5, "da_wiktionary_difficulty_limit": 5, "de_wiktionary_difficulty_limit": 5, "el_wiktionary_difficulty_limit": 5, "en_wiktionary_difficulty_limit": 5, "es_wiktionary_difficulty_limit": 5, "fi_wiktionary_difficulty_limit": 5, "fr_wiktionary_difficulty_limit": 5, "he_wiktionary_difficulty_limit": 5, "hr_wiktionary_difficulty_limit": 5, "it_wiktionary_difficulty_limit": 5, "ja_wiktionary_difficulty_limit": 5, "ko_wiktionary_difficulty_limit": 5, "lt_wiktionary_difficulty_limit": 5, "mk_wiktionary_difficulty_limit": 5, "nl_wiktionary_difficulty_limit": 5, "no_wiktionary_difficulty_limit": 5, "pl_wiktionary_difficulty_limit": 5, "pt_wiktionary_difficulty_limit": 5, "ro_wiktionary_difficulty_limit": 5, "ru_wiktionary_difficulty_limit": 5, "sl_wiktionary_difficulty_limit": 5, "sv_wiktionary_difficulty_limit": 5, "uk_wiktionary_difficulty_limit": 5, "zh_wiktionary_difficulty_limit": 5}']' returned non-zero exit status 1.

Called with args: (ParseJobData(book_id=148, book_path='F:\Calibre\Calibre Library\Steve Hays\A Case for Christ (148)\A Case for Christ - Steve Hays.azw3', mi=<calibre.ebooks.metadata.book.base.Metadata object at 0x000001BB73A08190>, book_fmt='AZW3', book_lang='en', useragent='WordDumb/3.31.4 (https://github.com/xxyzz/WordDumb)', plugin_path='C:\Users\cl930\AppData\Roaming\calibre\plugins\WordDumb.zip', spacy_model='en_core_web_trf', create_ww=True, create_x=True, asin='BBKMF8ZBLG', acr='A_Case_for_Christ', revision='07efc666', kfx_json=None, mobi_html=None, mobi_codec='utf-8'),) {'notifications': <queue.Queue object at 0x000001BB73A0AB50>, 'abort': <threading.Event at 0x1bb73a3b4d0: unset>, 'log': <calibre.utils.logging.GUILog object at 0x000001BB73A3B1D0>} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Users\cl930\AppData\Roaming\calibre\plugins\WordDumb.zip__main.py", line 37, in File "C:\Users\cl930\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 227, in create_files File "C:\Users\cl930\AppData\Roaming\calibre\plugins\WordDumb.zip\parse_job.py", line 753, in load_spacy File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\spacy__init__.py", line 6, in from .errors import setup_default_warnings File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\spacy\errors.py", line 3, in from .compat import Literal File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\spacy\compat.py", line 4, in from thinc.util import copy_array File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\thinc__init__.py", line 5, in from .config import registry File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\thinc\config.py", line 5, in from .types import Decorator File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\thinc\types.py", line 25, in from .compat import cupy, has_cupy File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\thinc\compat.py", line 35, in import torch File "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\torch\init__.py", line 141, in raise err OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\cl930\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12\torch\lib\shm.dll" or one of its dependencies.

xxyzz commented 6 months ago

pytorch is downgraded to 2.2.2: https://github.com/xxyzz/WordDumb/commit/f0a99d6af5c82eb6d6ff6f4a9437a902113b4afa

You can download the test plugin from here: https://github.com/xxyzz/WordDumb/actions/runs/8910601306

py930828 commented 6 months ago

Downgrade pytorch to 2.2.2 works.

xxyzz commented 5 months ago

pytorch 2.3.1 should fix the bug, I have updated pytorch in 650c896a485fa9c1740c74ebdb808e82941fe649.

wylker commented 5 months ago

I'm still having this issue with pytorch 2.3.1

e; this error actually went away on calibre restart with pytorch 2.3.1 but I'm having a different issue now - will verify and open it seperately

xxyzz commented 3 months ago

GPU models are not supported now.