AssertionError: Definition of spacy.ml.models.multi_task.Vocab is unexpectedly incomplete

[ZeeD]

I see errors while using mypy on any codebase that relies on spacy.

How to reproduce the behaviour

issue.py

import spacy

mypy.ini

[mypy]
strict = True

trying to run mypy gives me this:

>mypy issue.py
Traceback (most recent call last):
  File "c:\Program Files\Python38\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\Program Files\Python38\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\vito.detullio\Desktop\workspace-srm-cognitive\azure-serverless-functions_VENV\Scripts\mypy.exe\__main__.py", line 9, in <module>
  File "c:\users\vito.detullio\desktop\workspace-srm-cognitive\azure-serverless-functions_venv\lib\site-packages\mypy\__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy\main.py", line 87, in main
  File "mypy\main.py", line 165, in run_build
  File "mypy\build.py", line 179, in build
  File "mypy\build.py", line 254, in _build
  File "mypy\build.py", line 2697, in dispatch
  File "mypy\build.py", line 3021, in process_graph
  File "mypy\build.py", line 3138, in process_stale_scc
  File "mypy\build.py", line 2288, in write_cache
  File "mypy\build.py", line 1475, in write_cache
  File "mypy\nodes.py", line 313, in serialize
  File "mypy\nodes.py", line 3149, in serialize
  File "mypy\nodes.py", line 3083, in serialize
AssertionError: Definition of spacy.ml.models.multi_task.Vocab is unexpectedly incomplete

I'm on windows, with python 3.8.0 and the latest versions of spacy and mypy

python -mpip list
Package                Version     Editable project location
---------------------- ----------- ---------------------------------------------------------------------------------
[...]
mypy                   0.910
mypy-extensions        0.4.3
[...]
spacy                  3.2.0
spacy-alignments       0.8.4
spacy-legacy           3.0.8
spacy-loggers          1.0.1
spacy-streamlit        1.0.2
spacy-transformers     1.1.2

[svlandeg]

Hm, that's ... interesting. It feels like this might be a bug on our end, so I labeled it as such for now. I'm a bit unsure how this can pop up though. Just to confirm: you're not getting any errors when running mypy on just the spacy repo, right?

[ZeeD]

you mean on a clone of this repository? I'll try later

[svlandeg]

Yes exactly. We run mypy as part of the CI, and we don't get these kind of failures, so I'm trying to understand what's going on. I see two options: either something in your configuration / mypy version is resulting in the error, or the error happens only when spacy is used as a dependency in another project. Which is why it would be helpful if you'd try running mypy on the spacy repo by itself, so we can get to the bottom of this!

[ZeeD]

same machine, python/mypy interpreter, virtualenv, etc. No errors with mypy. Some errors with mypy strict but no "hard crash" with the AssertionError

>git clone https://github.com/explosion/spaCy.git
Cloning into 'spaCy'...
remote: Enumerating objects: 99714, done.
remote: Counting objects: 100% (2746/2746), done.
remote: Compressing objects: 100% (1468/1468), done.
remote: Total 99714 (delta 1825), reused 1910 (delta 1276), pack-reused 96968
Receiving objects: 100% (99714/99714), 181.15 MiB | 3.89 MiB/s, done.
Resolving deltas: 100% (74711/74711), done.
Updating files: 100% (1556/1556), done.
>cd spaCy
>mypy spacy
spacy\tests\pipeline\test_analysis.py:3: error: Library stubs not installed for "mock" (or incompatible with Python 3.8)
spacy\tests\matcher\test_phrase_matcher.py:3: error: Library stubs not installed for "mock" (or incompatible with Python 3.8)
spacy\tests\doc\test_underscore.py:2: error: Library stubs not installed for "mock" (or incompatible with Python 3.8)
spacy\tests\doc\test_underscore.py:2: note: Hint: "python3 -m pip install types-mock"
spacy\tests\doc\test_underscore.py:2: note: (or run "mypy --install-types" to install all missing stub packages)
spacy\tests\doc\test_underscore.py:2: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
spacy\tests\matcher\test_matcher_api.py:2: error: Library stubs not installed for "mock" (or incompatible with Python 3.8)
spacy\tests\matcher\test_dependency_matcher.py:5: error: Library stubs not installed for "mock" (or incompatible with Python 3.8)
Found 5 errors in 5 files (checked 740 source files)
>mypy --install-types
[...]
>mypy spacy
Success: no issues found in 740 source files
>mypy --strict spacy
[...]
Found 4550 errors in 451 files (checked 740 source files)
>

[mathematicalmichael]

+1, having this same problem.

currently just ignoring the entire file in which spacy is imported during type checks. I was able to allow it to pass by telling mypy to skip following imports on that file, but it still added so much time to type-checking that skipping the file entirely was preferable.

[jboynyc]

+1 I'm getting this too

[Ce11an]

+1, also getting this issue. Any update? :)

[polm]

I can't reproduce this with this config:

mypy==0.942
mypy-extensions==0.4.3
spacy==3.2.3

Python version 3.10.2, Linux.

I ran mypy --strict check.py where check.py is just import spacy, like the initial post.

If you are encountering this error, please share the versions of the related software you're using.

It's not really clear to us why this would be happening - as mentioned upthread, we run mypy on spaCy itself as part of our CI.

Some things that might be at issue:

• specific mypy version
• something to do with the Python version
• something else in the mypy config
• Windows?

[mathematicalmichael]

did some investigating against spacy==3.1.0 and spacy==3.2.0 and found the issue was with the mypy==0.910 version in the original issue.

docker run --rm -ti python:3.8.0 bash pip3 install spacy==3.2.0 mypy==0.910 mypy-extensions==0.4.3 echo "import spacy" > check.py mypy --strict check.py

fix was in the following version: pip install mypy==0.920 mypy --strict check.py

[polm]

Excellent, thank you for tracking that down! I was indeed able to reproduce the error by downgrading to 0.910.

I'll mark this as resolved, but if anyone is still having issues please let us know.

[github-actions[bot]]

This issue has been automatically closed because it was answered and there was no follow-up discussion.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.