dmypy crashes every 2nd run when trying to update unlreated and unchanged numpy.compat

aberres commented 3 years ago

Crash Report

When dmypy is run the second time it crashes when trying to update its internal state. No code has been changed between runs.

Traceback

Traceback (most recent call last):
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 229, in serve
    resp = self.run_command(command, data)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 272, in run_command
    return method(self, **data)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 331, in cmd_run
    return self.check(sources, is_tty, terminal_width)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 393, in check
    messages = self.fine_grained_increment_follow_imports(sources)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 637, in fine_grained_increment_follow_imports
    messages = fine_grained_manager.update([], to_delete)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/server/update.py", line 272, in update
    messages = self.manager.errors.new_messages()
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/errors.py", line 587, in new_messages
    msgs.extend(self.file_messages(path))
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/errors.py", line 575, in file_messages
    return self.format_messages(self.error_info_map[path], source_lines)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/errors.py", line 548, in format_messages
    source_line = source_lines[line - 1]
IndexError: list index out of range

To Reproduce

I do not yet have a stand alone repo, but collected some logs what happens.

The problematic call is simply

dmypy run -- --config-file mypy.ini -p flask_rest_json_api

In the config we set pretty=True which causes the error to show up.

When the crash happens, dmypy tries to update the error messages for the system wide installed numpy. The detected error is absolutely bogus. The file python3.9/site-packages/numpy/core/overrides.py has only 227 lines while multiple errors are detected between lines 2936 and 3363 (see log below). Hence the crash.

The thing is: We are not at all interested in these numpy errors. Numpy should not be a package errors are reported for. And the first run actually does not report any numpy errors.

The verbose logs from the second run. As we see numpy.compat is considered to be deleted. The integrity_problem log has beed added by myself.

LOG:  fine-grained increment: find_changed: 0.004s
changed []
new_files []
to_delete: [('email.mime', '/Users/berres/venv/xyz/lib/python3.9/site-packages/mypy/typeshed/stdlib/email/mime/__init__.pyi'), ('html', '/Users/berres/venv/xyz/lib/python3.9/site-packages/mypy/typeshed/stdlib/html/__init__.pyi'), ('numpy.compat', '/Users/berres/venv/xyz/lib/python3.9/site-packages/numpy/compat/__init__.py'), ('numpy.compat.py3k', '/Users/berres/venv/xyz/lib/python3.9/site-packages/numpy/compat/py3k.py')] <class 'list'>
LOG:  fine-grained: ==== update 'email.mime', 'html', 'numpy.compat', 'numpy.compat.py3k' ====
LOG:  fine-grained: previous targets with errors: ['flask_rest_json_api.resource.ResourceDetail.patch', 'flask_rest_json_api.resource.ResourceList.get']
LOG:  fine-grained: --- update single 'email.mime' ---
LOG:  fine-grained: delete module 'email.mime'
LOG:  fine-grained: triggered: ['<email.mime>', '<email.mime[wildcard]>']
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: email.mime in 0.001s - 3 left
LOG:  fine-grained: --- update single 'html' ---
LOG:  fine-grained: delete module 'html'
LOG:  fine-grained: triggered: ['<html.AnyStr>', '<html.escape>', '<html.unescape>', '<html>', '<html[wildcard]>']
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: html in 0.000s - 2 left
LOG:  fine-grained: --- update single 'numpy.compat' ---
LOG:  fine-grained: delete module 'numpy.compat'
LOG:  fine-grained: triggered: ['<numpy.compat.Path>', '<numpy.compat._inspect>', '<numpy.compat.asbytes>', '<numpy.compat.asbytes_nested>', '<numpy.compat.asstr>', '<numpy.compat.asunicode>', '<numpy.compat.asunicode_nested>', '<numpy.compat.basestring>', '<numpy.compat.bytes>', '<numpy.compat.contextlib_nullcontext>', '<numpy.compat.formatargspec>', '<numpy.compat.getargspec>', '<numpy.compat.getexception>', '<numpy.compat.integer_types>', '<numpy.compat.is_pathlib_path>', '<numpy.compat.isfileobj>', '<numpy.compat.long>', '<numpy.compat.npy_load_module>', '<numpy.compat.open_latin1>', '<numpy.compat.os_PathLike>', '<numpy.compat.os_fspath>', '<numpy.compat.pickle>', '<numpy.compat.py3k>', '<numpy.compat.sixu>', '<numpy.compat.strchar>', '<numpy.compat.unicode>', '<numpy.compat>', '<numpy.compat[wildcard]>']
LOG:  fine-grained: process: numpy.compat
LOG:  fine-grained: Can't find matching target for numpy.compat (stale dependency?)
LOG:  fine-grained: process: numpy.core.overrides
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: numpy.compat in 2.971s - 1 left
LOG:  fine-grained: --- update single 'numpy.compat.py3k' ---
LOG:  fine-grained: delete module 'numpy.compat.py3k'
LOG:  fine-grained: triggered: ['<numpy.compat.py3k.Path>', '<numpy.compat.py3k.abc>', '<numpy.compat.py3k.abc_ABC>', '<numpy.compat.py3k.asbytes>', '<numpy.compat.py3k.asbytes_nested>', '<numpy.compat.py3k.asstr>', '<numpy.compat.py3k.asunicode>', '<numpy.compat.py3k.asunicode_nested>', '<numpy.compat.py3k.basestring>', '<numpy.compat.py3k.bytes>', '<numpy.compat.py3k.contextlib_nullcontext>', '<numpy.compat.py3k.getexception>', '<numpy.compat.py3k.integer_types>', '<numpy.compat.py3k.io>', '<numpy.compat.py3k.is_pathlib_path>', '<numpy.compat.py3k.isfileobj>', '<numpy.compat.py3k.long>', '<numpy.compat.py3k.npy_load_module>', '<numpy.compat.py3k.open_latin1>', '<numpy.compat.py3k.os>', '<numpy.compat.py3k.os_PathLike>', '<numpy.compat.py3k.os_fspath>', '<numpy.compat.py3k.pickle>', '<numpy.compat.py3k.sixu>', '<numpy.compat.py3k.strchar>', '<numpy.compat.py3k.sys>', '<numpy.compat.py3k.unicode>', '<numpy.compat.py3k>', '<numpy.compat.py3k[wildcard]>']
LOG:  fine-grained: process: numpy.compat.py3k.asbytes_nested
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k.asbytes_nested (stale dependency?)
LOG:  fine-grained: process: numpy.compat.py3k.contextlib_nullcontext
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k.contextlib_nullcontext (stale dependency?)
LOG:  fine-grained: process: numpy.compat
LOG:  fine-grained: Can't find matching target for numpy.compat (stale dependency?)
LOG:  fine-grained: process: numpy.compat.py3k
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k (stale dependency?)
LOG:  fine-grained: process: numpy.compat.py3k.asunicode_nested
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k.asunicode_nested (stale dependency?)
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: numpy.compat.py3k in 2.855s - 0 left
LOG:  fine-grained: process target with error: flask_rest_json_api.resource.ResourceList.get
LOG:  fine-grained: process target with error: flask_rest_json_api.resource.ResourceDetail.patch
LOG:  fine-grained: triggered: []
prev errors {'flask_rest_json_api.resource.ResourceList.get', 'flask_rest_json_api.resource.ResourceDetail.patch'}
errors {'/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py': [<mypy.errors.ErrorInfo object at 0x10b347820>, <mypy.errors.ErrorInfo object at 0x10b3477c0>, <mypy.errors.ErrorInfo object at 0x10b347910>, <mypy.errors.ErrorInfo object at 0x10b347940>, <mypy.errors.ErrorInfo object at 0x10b347970>, <mypy.errors.ErrorInfo object at 0x10b3478e0>, <mypy.errors.ErrorInfo object at 0x10b3479d0>, <mypy.errors.ErrorInfo object at 0x10b347a30>, <mypy.errors.ErrorInfo object at 0x10b347a00>, <mypy.errors.ErrorInfo object at 0x10b347a60>], '/Users/berres/Devel/mpptool/backend/flask-rest-json-api/flask_rest_json_api/resource.py': [<mypy.errors.ErrorInfo object at 0x109fe00a0>, <mypy.errors.ErrorInfo object at 0x108fcc820>]}
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 2936 Class numpy.number has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 2936 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3088 Class numpy.integer has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3088 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3244 Class numpy.inexact has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3244 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3344 Class numpy.flexible has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3344 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3363 Class numpy.character has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3363 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
LOG:  fine-grained increment: update: 5.955s

Your Environment

Mypy version used: 0.920+dev.7576f659d42ee271c78e69d7481b9d49517a49f6

Mypy configuration options from mypy.ini (and other config files):

[mypy]
ignore_missing_imports = True
color_output = True
pretty = True
show_error_codes = True
check_untyped_defs = True
warn_unreachable = True
error_summary = True

Python version used: 3.9
Operating system and version: MacOS

aberres commented 3 years ago

No idea if this is intended or not, but when I compare the dmypy and the mypy output I notice something else:

dmypy checks way more files, while mypy only cares about the passed package. When I do not pass pretty=True instead of a crash I get the bogus error messages for numpy as errors:

First run

> dmypy run -- --config-file mypy.ini -p flask_rest_json_api
backend/flask-rest-json-api/flask_rest_json_api/resource.py:146: error: Need type annotation for "schema_kwargs"  [var-annotated]
backend/flask-rest-json-api/flask_rest_json_api/resource.py:272: error: Need type annotation for "schema_kwargs"  [var-annotated]
Found 2 errors in 1 file (checked 405 source files)

Second run

> dmypy run -- --config-file mypy.ini -p flask_rest_json_api
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:2936: error: Class numpy.number has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:2936: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3088: error: Class numpy.integer has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3088: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3244: error: Class numpy.inexact has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3244: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3344: error: Class numpy.flexible has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3344: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3363: error: Class numpy.character has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3363: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
backend/flask-rest-json-api/flask_rest_json_api/resource.py:146: error: Need type annotation for "schema_kwargs"  [var-annotated]
backend/flask-rest-json-api/flask_rest_json_api/resource.py:272: error: Need type annotation for "schema_kwargs"  [var-annotated

Pure mypy

> mypy -p flask_rest_json_api --config mypy.ini
backend/flask-rest-json-api/flask_rest_json_api/resource.py:146: error: Need type annotation for "schema_kwargs"  [var-annotated]
backend/flask-rest-json-api/flask_rest_json_api/resource.py:272: error: Need type annotation for "schema_kwargs"  [var-annotated]
Found 2 errors in 1 file (checked 24 source files)

aberres commented 3 years ago

I am trying to come up with a minimal reproduction, but it is not really obvious for me, why the numpy error is raised here.

What I know is that I have a dependency chain flask_rest_json_api->OtherPackage->numpy (not nice, but...).

When trying to reproduce I noticed something else which might be related: dmypy has sometimes problems to only check for errors in the specified package.

Repro: 1) Install a package which has no mypy errors 2) Start dmypy server like dmypy daemon -- -p my_package --verbose 3) Trigger check dmypy run -- -p my_package --verbose - everything is fine 4) Add an import which has mypy errors when checked individually. E.g. import marshmallow. 5) Rerun dmypy run -- -p my_package --verbose and observe a bunch of errors directly in marshmallow. 6) Restart dmypy and observe that errors are gone.

JelleZijlstra commented 3 years ago

Does this reproduce on master? It sounds like the kind of issue that #10937 recently fixed.

aberres commented 3 years ago

@JelleZijlstra Yeps, could reproduce with master. Only dmymy has the problem, could not crash things with mypy only.

bobwhitelock commented 3 years ago

I'm getting the same issues the second time I run dmypy run ., after just updating to numpy 1.21.2 (I get the same extra errors reported for numpy, and I get that crash if I add pretty = True to my config). Using Python 3.8 and mypy 0.790.

bobwhitelock commented 2 years ago

I'm getting the same issues the second time I run dmypy run ., after just updating to numpy 1.21.2 (I get the same extra errors reported for numpy, and I get that crash if I add pretty = True to my config). Using Python 3.8 and mypy 0.790.

I no longer see this issue on numpy 1.22.3 and Python 3.9.7, so possibly one of these updates fixed the issue. Still using the same version of mypy.

Although if a change in numpy can cause the daemon to crash like this it's possible there's still an underlying issue that needs to be fixed, but this no longer seems as important if it has cleared up

python / mypy

dmypy crashes every 2nd run when trying to update unlreated and unchanged numpy.compat #11009