Linux: spacy installed in venv fails due to pydantic import

[zeroeightysix]

Probably related to #140

I've installed anki-bin from the AUR. It shares the same python version as is installed system-wide:

Anki (anki-bin 24.04.1-2 from the AUR):

Version ⁨24.04.1 (ccd9ca1a)⁩ Python 3.12.3 Qt 6.7.1 PyQt 6.7.0

~ % /usr/bin/python --version
Python 3.12.3

I'm trying to set up spaCy for danish, and have followed the setup instructions for that from the guide, creating a venv with spacy installed in it in the addons directory. That works:

ridan@luna ~/.local/share/Anki2/addons21 % ls
472573498  spacyenv
ridan@luna ~/.local/share/Anki2/addons21 % source spacyenv/bin/activate
ridan@luna ~/.local/share/Anki2/addons21 % python -c "import spacy.util"
# (this does not error)

Yet, attempting to list the generators in Anki fails with the following error:

Anki 24.04.1 (ccd9ca1a) (src) (ao)
Python 3.12.3 Qt 6.7.1 PyQt 6.7.0
Platform: Linux-6.9.2-arch1-1-x86_64-with-glibc2.39

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/aqt/__init__.py", line 146, in open
    instance = creator(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ridan/.local/share/Anki2/addons21/472573498/generators_window.py", line 71, in __init__
    self._morphemizers: list[Morphemizer] = morphemizer.get_all_morphemizers()
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ridan/.local/share/Anki2/addons21/472573498/morphemizer.py", line 66, in get_all_morphemizers
    for spacy_model in spacy_wrapper.get_installed_models():
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ridan/.local/share/Anki2/addons21/472573498/spacy_wrapper.py", line 354, in get_installed_models
    import spacy.util  # pylint:disable=import-outside-toplevel
    ^^^^^^^^^^^^^^^^^
  File "/home/ridan/.local/share/Anki2/addons21/spacyenv/lib/python3.12/site-packages/spacy/__init__.py", line 6, in <module>
    from .errors import setup_default_warnings
  File "/home/ridan/.local/share/Anki2/addons21/spacyenv/lib/python3.12/site-packages/spacy/errors.py", line 3, in <module>
    from .compat import Literal
  File "/home/ridan/.local/share/Anki2/addons21/spacyenv/lib/python3.12/site-packages/spacy/compat.py", line 4, in <module>
    from thinc.util import copy_array
  File "/usr/lib/python3.12/site-packages/thinc/__init__.py", line 5, in <module>
    from .config import registry
  File "/usr/lib/python3.12/site-packages/thinc/config.py", line 2, in <module>
    import confection
  File "/usr/lib/python3.12/site-packages/confection/__init__.py", line 11, in <module>
    from pydantic.main import ModelMetaclass
ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main' (/usr/lib/python3.12/site-packages/pydantic/main.py)

===Add-ons (active)===
(add-on provided name [Add-on folder, installed at, version, is config changed])
AnkiMorphs ['472573498', 2024-05-18T19:57, 'None', mod]

===IDs of active AnkiWeb add-ons===
472573498

===Add-ons (inactive)===
(add-on provided name [Add-on folder, installed at, version, is config changed])

I also tried installing the version of pydantic listed in #140 (python -m pip install pydantic==1.10.14) with no luck.

The only way I can run anki with anki-morph is by installing all of anki's required dependencies into the venv (qt5/6+webengine) and launching the entire application from the venv.

System

Arch linux 6.9.2-arch1-1

[mortii]

That sucks, I wish there were fewer spaCy related problems :/

I also tried installing the version of pydantic listed in https://github.com/mortii/anki-morphs/issues/140 (python -m pip install pydantic==1.10.14) with no luck.

Did you run this command inside the venv? What is the version of your system wide pydantic package?

The confection package that is throwing the error has this requirement:

pydantic>=1.7.4,!=1.8,!=1.8.1,<3.0.0

but that range could be be poorly defined for all I know :man_shrugging:

[zeroeightysix]

Did you run this command inside the venv? What is the version of your system wide pydantic package?

I did. Here's the versions:

ridan@luna ~/.local/share/Anki2/addons21 % source spacyenv/bin/activate
ridan@luna ~/.local/share/Anki2/addons21 % python -c "import pydantic; print(pydantic.__version__)"
1.10.14
ridan@luna ~/.local/share/Anki2/addons21 % deactivate
ridan@luna ~/.local/share/Anki2/addons21 % python -c "import pydantic; print(pydantic.__version__)"
2.7.1

The confection package that is throwing the error has this requirement:

pydantic>=1.7.4,!=1.8,!=1.8.1,<3.0.0

but that range could be be poorly defined for all I know 🤷‍♂️

Seems like it :slightly_frowning_face: I can't downgrade my system-wide pydantic, either, because other packages depend on it

[mortii]

I can't downgrade my system-wide pydantic, either, because other packages depend on it

Understandable.

I'm on ubuntu, I don't have pydantic installed globally and my spacyenv has pydantic 2.7.2 and it works just fine, surprisingly...

The only way I can run anki with anki-morph is by installing all of anki's required dependencies into the venv (qt5/6+webengine) and launching the entire application from the venv.

does this also work when the venv has the latest pydantic version?

[zeroeightysix]

The only way I can run anki with anki-morph is by installing all of anki's required dependencies into the venv (qt5/6+webengine) and launching the entire application from the venv.

does this also work when the venv has the latest pydantic version?

Yep, strangely enough, it does. I tried both 2.7.1 (what my system has) and 2.7.2 (latest) and they work in the venv.

[mortii]

Yep, strangely enough, it does. I tried both 2.7.1 (what my system has) and 2.7.2 (latest) and they work in the venv.

That's wild. Maybe this was your initial setup, but If you set the venv pydantic version to be the same as the system version, and you start Anki normally outside the venv, does it still cause an exception?

[zeroeightysix]

I'm not home, so I tried reproducing the exact same setup on my laptop (also arch) but actually failed to reproduce the error. So, looks like something's awry on my desktop!

In a few hours, I'll see if I can spot the difference between the two.

[zeroeightysix]

Okay, the difference was that python-pydantic wasn't installed on my laptop. Turns out it was unnecessary on my desktop as well (orphaned), and deleting it did fix the problem.

Regarding your question, setting the venv pydantic == system pydantic did not fix the issue when launching outside of the venv.

Not sure where that takes us - seems like magic to me. Same versions, but only breakage if it's using the system-wide one; maybe the arch package is lying about itself? Build parameters? I don't have an idea :slightly_frowning_face:

[mortii]

Super weird. Maybe the package got broken when it was orphaned? Did you try to do a clean reinstall of the system-wide package?

[zeroeightysix]

Okay, looks like it's the python-thinc package breaking (dependant of pydantic, also orphaned).

Installing python-pydantic: no problems Installing python-confection: no problems Installing python-thinc: import thinc fails with our pydantic import error

From the AUR page, it's only required by python-spacy, so I might've had that installed earlier for another project? Either way, seems like this is a packaging issue, so feel free to close.

[mortii]

I want to add some info about this in the "Potential problems" section. What should we write there in your opinion? The AUR python-spacy / python-thinc packages are broken and have to be uninstalled?

[zeroeightysix]

I wouldn't call them broken on the guide page, because other than what we're observing I don't know what the genuine problem is.

Just mention that people have had troubles with anki-morphs when python-spacy and or python-thinc is installed, and recommend trying to uninstall those packages & try again?

[mortii]

Done :)

Thanks you so much for all the help!

[zeroeightysix]

Thank you for the speedy replies & help as well!

Love this project, so keep it going :slightly_smiling_face:

And happy learning!

[ashprice]

Hmm yeah I think I had issues with python-thinc before, but I don't recall details. FWIW, I have pydantic 2.7.1 and thinc 8.2.3, I'm pretty sure my (bad) solution was --overwrite-system-packages with pip.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.