mortii / anki-morphs

A MorphMan fork rebuilt from the ground up with a focus on simplicity, performance, and a codebase with minimal technical debt.
https://mortii.github.io/anki-morphs/
Mozilla Public License 2.0
47 stars 6 forks source link

SpaCy models not showing in Anki #197

Closed syfgk closed 3 months ago

syfgk commented 3 months ago

(I'm new to creating issues in Github so sorry if my formatting is off. Let me know what information is needed/helpful for debugging and I can provide that — thanks for working on such an incredible project!)

Describe the bug

I'm on MacOS. I followed the instructions on the Github page but I still don't see the SpaCy models in Anki.

Here are a list of things that I've tried:

One thing also to note is:

When I type "python3 --version" the terminal returns python 3.11. However, when I type "python --version", I get python 3.9.

So when I was following your instructions, I made sure to specify "python" and not "python3". However doing this didn't help either.

Steps to reproduce the behavior

Expected behavior

Screenshots

image

My setup

Additional context

mortii commented 3 months ago

Sorry to hear that it's not working as expected--I wish there was an easier way to do it so this kind of problem wouldn't occur.

When I type "python3 --version" the terminal returns python 3.11. However, when I type "python --version", I get python 3.9.

This might be the heart of the issue. What do you get if you type this:

python3.9 --version

Adding the .9 to the command might disambiguate the python version you are using.

You can double check if that is the problem by going into the virtual environment you created and look at the python version it is using:

cd ~/Library/Application\ Support/Anki2/addons21
. spacyenv/bin/activate
python --version
syfgk commented 3 months ago

Thanks for the help! I was able to figure it out. Here's what worked for me: instead of using python3.9 or python in the commands below, I used python3.

cd ~/Library/Application\ Support/Anki2/addons21
python3.9 -m pip install --upgrade pip virtualenv
python3.9 -m venv spacyenv
. spacyenv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install --upgrade spacy six

The issue before was that SpaCy was being installed on my root and not in the virtual environment. So Anki Morphs was never actually picking up the SpaCy package (as Anki only looks for Python packages in its own folders). I was able to tease this out by looking at the terminal. As SpaCy was installing, there was a line that said something along the lines of:

defaulting to user installation because normal site-packages is not writeable ("normal site-packages" meaning the site-packages folder in the virtual env that sits in the Anki profile folder)

mortii commented 3 months ago

Hmm, that is very weird, and I don't think your solution would work unless you have some custom components. Could you provide some additional info so I can get to the bottom of this issue? I want to know the following:

  1. Do you see the installed spaCy models in the AnkiMorphs settings?
  2. What python version is your Anki using?
    (go to "help" -> "about" and you should see something like "Python 3.9.15 Qt 6.6.1 PyQt 6.6.1")
  3. You mentioned:

    When I type "python3 --version" the terminal returns python 3.11.

    When you type python3 --version in a fresh terminal do you still get 3.11 or did you modify the path or something to change the result?

  4. What is the output when you do this?
    cd ~/Library/Application\ Support/Anki2/addons21
    . spacyenv/bin/activate
    python --version
syfgk commented 3 months ago

Do you see the installed spaCy models in the AnkiMorphs settings?

I do now! This is after using python3 instead though. Here's what I currently see in AnkiMorphs:

image

What python version is your Anki using?

Python 3.9.15 Qt 6.5.3 PyQt 6.5.3

When you type python3 --version in a fresh terminal do you still get 3.11 or did you modify the path or something to change the result?

Python 3.11.0 — this is with no modifications.

What is the output when you do this? Python 3.9.6 (and then the row immediately after shows my terminal is now in the venv "spacyenv").

mortii commented 3 months ago

@syfgk thank you so much!

With that additional info I can confidently say that it shouldn't have worked. The only explanation I can think of is that you kept using the same terminal after you had already activated the virtual environment, and therefore the the command python3 returned 3.9.6 instead of 3.11.0

If that is the case, then I'm guessing the original problem was caused by the fact that you used the command:

python3 -m venv spacyenv

instead of the command that is in the guide:

python3.9 -m venv spacyenv

Could you do me a huge favor and redo everything from scratch? I really want to find out if the instructions in the guide are wrong--that would obviously be pretty bad.

So basically do this:

  1. delete the spacyenv folder that is in ~/Library/Application\ Support/Anki2/addons21
  2. open a fresh terminal
  3. copy paste the commands found in the guide

If any of the commands you copy-pasted fails, then we will have narrowed down the problem.

jahnke commented 3 months ago

Hi. I have the exact same problem. I had to change "py -3.9" for "python3.9". After the change, I runned the following on a freshly open terminal after removing the spacyenv folder:

cd %HOMEPATH%\AppData\Roaming\Anki2\addons21
python3.9 -m pip install --upgrade pip virtualenv
python3.9 -m venv spacyenv
spacyenv\Scripts\activate
python3.9 -m pip install --upgrade pip setuptools wheel
python3.9 -m pip install --upgrade spacy six
python3.9 -m spacy download de_core_news_lg
deactivate

I only see the option for languages with spaces. There's no option to use "spaCy".

I don't consider this to be such a huge issue because AnkiMorph is still helping me a lot to find good sentences to learn, but I would be very glad if I could figure it our how to use it to is full capabilities. Thanks!

syfgk commented 3 months ago

I deleted the spacyenv folder. As expected this removed the SpaCy model:

image

The first line in the help guide says I need to type python3.9 --version; however, when I do this in a fresh terminal, I get this error: zsh: command not found: python3.9

If I typepython3 --version, I get: Python 3.11.0 If I type python --version, I get: Python 3.9.6

I did both of these in a fresh terminal btw.

syfgk commented 3 months ago

Ah; after deleting SpaCy, I can't get it to work again. To your earlier point, simply replacing python3.9 with python doesn't actually work. SpaCy isn't installing in the spacyenv folder.

image

Not sure what's going on...

syfgk commented 3 months ago

Got it working again!

image

Here's what I did: first I ran these commands:

cd ~/Library/Application\ Support/Anki2/addons21
python -m pip install --upgrade pip virtualenv
python -m venv spacyenv
. spacyenv/bin/activate

Then after I activated spacyenv, I did this:

python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install --upgrade spacy six
python3 -m spacy download [zh_core_web_lg](https://spacy.io/models/zh#zh_core_web_lg)

So basically I replaced python3.9 with python3.

The issue before was that whenever I tried using python3.9 I always got this message in the terminal:

Defaulting to user installation because normal site-packages is not writeable

mortii commented 3 months ago

The first line in the help guide says I need to type python3.9 --version; however, when I do this in a fresh terminal, I get this error: zsh: command not found: python3.9

@syfgk aha, how did you install python 3.9? It must be missing from your path.

Hi. I have the exact same problem. I had to change "py -3.9" for "python3.9".

@jahnke you are using windows right? What happens when you run py -3.9 --version?

jahnke commented 3 months ago

Hi, I don't have the command "py." I installed Python 3.9 using the Windows Installer / Software Manager (Sorry, I am not a Windows user, I don't know the exact name of the software but I can check it if its necessary). I can only access python by running python3.9. Is the command just "py" associated to something specific that I might be able to install?

mortii commented 3 months ago

I installed Python 3.9 using the Windows Installer / Software Manager

Apparently there is a known issue with installation from the Microsoft Store:

Because of restrictions on Microsoft Store apps, Python scripts may not have full write access to shared locations such as TEMP and the registry. Instead, it will write to a private copy. If your scripts must modify the shared locations, you will need to install the full installer.

I don't know if that comes into effect here though.

I can only access python by running python3.9. Is the command just "py" associated to something specific that I might be able to install?

It seems the aliases are different depending on the method of installation. I have a vague memory of there being some important difference between them, but It's been a long time since I made the guide for window, so I don't remember off the top of my head.

what output do you get when running python3.9 --version ?

jahnke commented 3 months ago

Dear @mortii,

I just got back home and, on my personal computer, following exactly the installation steps from the website, AnkiMorph works just fine. The issue is certainly caused by the python version from the Microsoft Store. It is worth it to understand how to solve such an edgy case? If you think it is, when I am back at the office, I can get the output you asked and give you more information, if you need.

Cheers!

mortii commented 3 months ago

I just got back home and, on my personal computer, following exactly the installation steps from the website, AnkiMorph works just fine.

Great! :)

The issue is certainly caused by the python version from the Microsoft Store. It is worth it to understand how to solve such an edgy case? If you think it is, when I am back at the office, I can get the output you asked and give you more information, if you need.

I think I'll probably just add a warning about it the guide instead of trying to debug it further. Thanks for the offer though!

syfgk commented 3 months ago

how did you install python 3.9? It must be missing from your path.

Ah, I don't recall unfortunately. How should I add it to my path?

mortii commented 3 months ago

Ah, I don't recall unfortunately. How should I add it to my path?

If you don't have the installer do it for you then it can be pretty complicated. Since you already got it to work, and you now know that your python and python3.9 commands are equivalent, it is probably not worth it to update the path at this point.

I added this part to the guide to hopefully prevent this from happening in the future:

Note: If you install Python 3.9 in a different way than the instructions below, then you might encounter important differences that could prevent you from accessing the spaCy morphemizers in Anki.

Thanks for all the feedback! Let me know if you guys have any more :)

syfgk commented 3 months ago

Thank you!

github-actions[bot] commented 3 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.