AssertionError: Internal error: method must be called on parsed file only

[daisuke834]

Bug Report

When I run mypy==0.991 in parallel, the following error randomly happens.

Traceback (most recent call last):
  File "/home/my_name/.pyenv/versions/3.9.16/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/my_name/.pyenv/versions/3.9.16/lib/python3.9/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "mypy/main.py", line 95, in main
  File "mypy/main.py", line 174, in run_build
  File "mypy/build.py", line 193, in build
  File "mypy/build.py", line 276, in _build
  File "mypy/build.py", line 2903, in dispatch
  File "mypy/build.py", line 3284, in process_graph
  File "mypy/build.py", line 3365, in process_fresh_modules
  File "mypy/build.py", line 2109, in fix_cross_refs
AssertionError: Internal error: method must be called on parsed file only

To Reproduce

I cannot identify to reproduce this problem because this error randomly happens (2-3% probabiliy), but I observed this problem only when I ran mypy in parallel on multiple files.

Your Environment

• Mypy version used: mypy==0.991
• Mypy command-line flags: mypy --follow-imports silent --config-file pyproject.toml path/to/foo.py
• Mypy configuration options from mypy.ini (and other config files): pyproject.toml

  [tool.mypy]
  ignore_missing_imports = true
  [[tool.mypy.overrides]]
  module = "third_party.*"
  ignore_errors = true
  [[tool.mypy.overrides]]
  module = "requests.*"
  ignore_missing_imports = true

• Python version used:3.9.16

[Alex031544]

Same here... It seems, that it occurs on files without findings.

E.g.: I put a faulty code in a file, where I assume the source looks fine:

a = dict()
for d in a:
    print(d)

... and the AssertionError disappears! But this might be only the half truth as some files pass anyhow. Maybe this is something in conjunction with the cache.

---

I found out, that this issue occurs in dependency with the cache. If several files are checked this leads to trouble. Using separate cache dirs helped.

[hauntsaninja]

@Alex031544 it seems your issue might be different from OP's, since it sounds like you're only running one mypy command at a time. If you can provide enough details that we could reproduce your issue, that would be much appreciated!

[Alex031544]

@Alex031544 it seems your issue might be different from OP's, since it sounds like you're only running one mypy command at a time. If you can provide enough details that we could reproduce your issue, that would be much appreciated!

That's how I call mypy as part of one unit test with one subcase per file - Therefore I am running several mypy tests parrallel:

import os
import glob
import multiprocessing
import re
import shutil
import subprocess
import unittest
from dataclasses import dataclass
from typing import List

@dataclass
class Result:
    return_val: int
    stdout: str
    stderr: str

class StaticTest(unittest.TestCase):

    mypy_cache_based_dir = '.mypy_cache'

    def _worker_mypy(self, filename: str, results: dict):
        print('mypy on ' + filename)
        cache_dir = self.mypy_cache_based_dir + '/' + filename
        cache_dir = re.sub(r'[^\w./]', '_', cache_dir)

        call: List[str] = ['.venv/bin/mypy'] \
                          + self.mypy_opts \
                          + ['--cache-dir %s' % cache_dir] \
                          + [filename]
        p = subprocess.Popen(' '.join(call),
                             shell=True,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
        stdout, stderr = p.communicate()
        p.wait()
        results[filename] = Result(p.returncode, stdout.decode(), stderr.decode())

    def test_run_mypy_tests(self):
        """
        Run MyPy on all files in search paths
        """

        os.mkdir(self.mypy_cache_based_dir)

        jobs = []
        manager = multiprocessing.Manager()
        results = manager.dict()
        for test_file in self.test_files:
            p = multiprocessing.Process(target=self._worker_mypy, args=(test_file, results,))
            jobs.append(p)
            p.start()

        for p in jobs:
            p.join()

        for key, result in results.items():
            with self.subTest(msg=f'mypy check on {key}'):
                self.assertEqual(0, result.return_val,
                                 f'mypy failed on {key}\n\n{result.stdout}\n\n{result.stderr}')
        shutil.rmtree(self.mypy_cache_based_dir, ignore_errors=True)

    def __init__(self, *args, **kwargs) -> None:
        self.pkgname: str = "browse"
        super(StaticTest, self).__init__(*args, **kwargs)
        my_env = os.environ.copy()
        self.pypath: str = my_env.get("PYTHONPATH", os.getcwd())

        self.mypy_opts: List[str] = ['--config-file .mypyrc']

        search_paths = ['pylib', 'pytest']
        self.test_files: List[str] = []
        for spath in search_paths:
            self.test_files.extend(glob.iglob(f'{os.getcwd()}/{spath}/**/*.py', recursive=True))

if __name__ == '__main__':
    unittest.main()

Now for each call respectively file under test, a unique path is created below '.mypy_cache'. The cache folder will be deleted when finished at once.

[daisuke834]

This issue was reproduced with 1.0.0 as well.

[gresavage]

I was consistently having this issue as well with mypy 0.993 when running as a pre-commit hook. As with @Alex031544 it was only occurring on files without findings.

I upgraded to 1.4.1 and it went away. Not sure of which versions between 1.0.0 (per @daisuke834) and 1.4.1 also fix the issue.

[gresavage]

I was consistently having this issue as well with mypy 0.993 when running as a pre-commit hook. As with @Alex031544 it was only occurring on files without findings.

I upgraded to 1.4.1 and it went away. Not sure of which versions between 1.0.0 (per @daisuke834) and 1.4.1 also fix the issue.

nevermind - I am still having the issue in 1.4.1 :face_exhaling:

[gresavage]

Sorry to bombard the thread - I am using a conda (specifically mamba) environment and found the following mypy info:

mypy                      1.4.1           py310h2372a71_0    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge

I then tried force installing mypy==1.4.1 from pip and retried. Lo and behold, the errors went away. So I wonder if it might be localized to the distributions on conda-forge???

[JelleZijlstra]

Possibly related to whether or not mypy is compiled with mypyc?

[lucemia]

As for me, it seems like the problem is connected to --install-types.

[LukasJerabek]

We experience similar issues with the same traceback, in our case it happens only if we add MYPYPATH=./src (which is our root source codes directory) in front of mypy call. When we remove it, problems disappear.

[peterhessey]

I was also having this issue when using mypy in a pre-commit hook with repo: local. Adding require_serial: true to the hook fixed this for me!

[LukasJerabek]

Our case was fixed by removing sqlalchemy from plugins from .mypy.ini from this: plugins = pydantic.mypy, sqlalchemy.ext.mypy.plugin to this: plugins = pydantic.mypy

[sshane]

Still occurring with parallel pre-commit==3.7.0 and mypy==1.9.0 every other run on a large project.

pyproject.toml:

[tool.mypy]
python_version = "3.11"

.pre-commit-config.yaml:

repos:
-   repo: local
    hooks:
    -   id: mypy
        name: mypy
        entry: mypy
        language: system
        types: [python]

[Ryang20718]

still occurring with mypy 1.10.0

[embray]

I have seen this recently as well in parallelized tests that are calling mypy. I think simply put it is not intended to be able to run in parallel--the problem occurs most likely when one process is writing or updating a cached MypyFile while the other process is trying to read the same file. If the latter gets a partially written (invalid) file here, this error can occur.

This could possibly be improved on the mypy side by locking around access to these files, but it might be some additional complexity/performance drag for minimal benefit to normal use cases. My approach to fixing it might be to just put a lock file around the tests that call mypy.

[KapJI]

Got this on v1.13.0. Adding require_serial: true solves it.