Dependencies conflict when installing en-core-web-trf

afarchant commented 9 months ago

Checkboxes

[X] I have read the document at xxyzz.github.io/WordDumb.
[X] I have not found similar issue or disscussion at GitHub.
[X] Reboot doesn't fix the problem.

Describe the bug

There seems to be a dependencies conflict when attempting to install en-core-web-trf (when CUDA is enabled). It does work when not using CUDA like when using the en-core-web-md model. Only during the installation of en-core-web-trf am I having issues. I tried multiple times, installing, uninstalling, deleting the worddumb-libs-py3.12 folder and reinstalling with no success.

Operating System name and version

Windows 11

Python version

3.12

calibre version

6.27

WordDumb plugin version

3.29.7

Error message

calibre, version 6.27.0 (win32, embedded-python: True)
Tonnerre de Brest!: An error occurred, please copy error message then report bug at GitHub.

Starting job: Generating Word Wise and X-Ray for Jane Eyre 
Job: "Generating Word Wise and X-Ray for Jane Eyre" failed with error: 
Traceback (most recent call last):
  File "calibre\gui2\threaded_jobs.py", line 82, in start_work
  File "calibre_plugins.worddumb.parse_job", line 170, in do_job
  File "calibre_plugins.worddumb.deps", line 53, in install_deps
  File "calibre_plugins.worddumb.deps", line 151, in pip_install
  File "calibre_plugins.worddumb.utils", line 55, in run_subprocess
  File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', '-m', 'pip', '--disable-pip-version-check', 'install', '-U', '-t', 'C:\\Users\\Chant\\AppData\\Roaming\\calibre\\plugins\\worddumb-libs-py3.12', '--no-user', 'https://github.com/explosion/spacy-models/releases/download/en_core_web_trf-3.7.2/en_core_web_trf-3.7.2-py3-none-any.whl']' returned non-zero exit status 1.

Called with args: (ParseJobData(book_id=50, book_path='C:\\Users\\Chant\\Calibre Library\\Charlotte Bronte\\Jane Eyre (50)\\Jane Eyre - Charlotte Bronte.mobi', mi=<calibre.ebooks.metadata.book.base.Metadata object at 0x00000286AD2B0940>, book_fmt='MOBI', book_lang='en', useragent='WordDumb/3.29.7 (https://github.com/xxyzz/WordDumb)', plugin_path=WindowsPath('C:/Users/Chant/AppData/Roaming/calibre/plugins/WordDumb.zip'), spacy_model='en_core_web_trf', create_ww=True, create_x=True, asin='BBAX3MS5JK', acr='Jane_Eyre', revision='d18e20bc', update_asin=False, kfx_json=None, mobi_html=b'<html><head><guide><reference type="toc" title="Table of Contents" filepos=0000001085 /><reference type="copyright-page" ) 
...
*** [Removed Book Content to fit in comment box] ***
...
{'notifications': <queue.Queue object at 0x00000286AD2B10C0>, 'abort': <threading.Event object at 0x00000286AD2B0AC0>, 'log': <calibre.utils.logging.GUILog object at 0x00000286AD2B0880>} 
ERROR: Cannot install en-core-web-trf because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Plugin settings and reproduce steps

０.Check the box that says ”Run spaCy with GPU(requires CUDA)

Change CUDA version to "CUDA 11.8"
spaCy model size set to "Medium" (tried Large also)
Fetch X-Ray people descriptions from Wikipedia/Fandom also checked.

Generated files, screenshots or videos

xxyzz commented 9 months ago

The error message doesn't say which dependencies are conflicting? I guess it's because some dependencies don't support Python 3.12, please try 3.11.

dbdness commented 9 months ago

The error message doesn't say which dependencies are conflicting? I guess it's because some dependencies don't support Python 3.12, please try 3.11.

Correct, it seems like torch is the conflicting package and that it doesn't support Python 3.12. I got the following information when trying to run the WordDumb pip install command manually.

ERROR: Cannot install en-core-web-trf because these package versions have conflicting dependencies. The conflict is caused by: spacy-curated-transformers 0.2.1 depends on torch>=1.12.0 spacy-curated-transformers 0.2.0 depends on torch>=1.12.0

How can I make Worddumb use Python 3.11 while having 3.12 installed as well? I have tried setting my global Python version to 3.11 in various ways, but WordDumb still attempts to pip install to the following target

(...)\AppData\Roaming\calibre\plugins\worddumb-libs-py3.12

xxyzz commented 9 months ago

I add a new configuration option to set the Python interpreter path: https://github.com/xxyzz/WordDumb/commit/3304d23f4bc9651f7cffefbf0638de4167c99e20

dbdness commented 9 months ago

Thank you so much, really appreciate all the work you do with this fantastic plugin. Will it be available as a new release, updateable via calibre plugin explorer?

Happy new year!

xxyzz commented 9 months ago

You could download the test plugin from GitHub Actions Artifact: https://github.com/xxyzz/WordDumb/actions/runs/7370508662

If the commit fixes the error and hopefully doesn't have new bugs I'll create a release.

xxyzz commented 8 months ago

v3.30.0 is released, close the issue now.

xxyzz / WordDumb