search

davidhalter / jedi

Awesome autocompletion, static analysis and refactoring library for python
http://jedi.readthedocs.io
Other
5.8k stars 508 forks source link

RecursionError on numpy 2.0.x when completing `numpy.finfo` and reading `type` #2020

Open niloc132 opened 3 months ago

niloc132 commented 3 months ago

Steps to reproduce:

Install latest jedi (currently 0.19.1) and numpy (verified with 2.0.0 and 2.0.1)

$ python -m jed
$ python -m venv jedi-numpy
$ source jedi-numpy/bin/activate
$ pip install jedi numpy
$ python

Verify versions

>>> import jedi
>>> jedi.__version__
'0.19.1'
>>> import numpy
>>> numpy.__version__
'2.0.1'

Try to read the type of numpy.finfo:

>>> from jedi import Interpreter
>>> items = Interpreter('import numpy\nnumpy.finfo', [locals()]).complete()
>>> items
[<Completion: finfo>]
>>> items[0].type
Traceback (most recent call last):
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/api/completion_cache.py", line 21, in _get_from_cache
    return _cache[module_name][name][number]
KeyError: 'numpy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 651, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 238, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 356, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
... trimmed 1876 lines here, the same 14 as above, still repeated a few times below...
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 651, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 238, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 356, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 651, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 238, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 356, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 651, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 238, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 356, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 651, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 238, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 356, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 651, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 238, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 356, in py__doc__
    return _merge_name_docs(names)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 24, in _merge_name_docs
    doc += name.py__doc__()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/names.py", line 645, in py__doc__
    names = convert_names(names, prefer_stub_to_compiled=False)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/gradual/conversion.py", line 154, in convert_names
    return _try_stub_to_python_names(
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/utils.py", line 16, in wrapper
    return list(func(*args, **kwargs))
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/gradual/conversion.py", line 73, in _try_stub_to_python_names
    values = convert_values(name.infer(), ignore_compiled=prefer_stub_to_compiled)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/gradual/conversion.py", line 168, in convert_values
    return ValueSet.from_sets(
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 430, in from_sets
    for set_ in sets:
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/gradual/conversion.py", line 169, in <genexpr>
    _stub_to_python_value_set(stub_value, ignore_compiled=ignore_compiled)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/gradual/conversion.py", line 36, in _stub_to_python_value_set
    values = _infer_from_stub(stub_module_context, qualified_names, ignore_compiled)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/gradual/conversion.py", line 60, in _infer_from_stub
    non_stubs = non_stubs.py__getattribute__(name)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 496, in py__getattribute__
    return ValueSet.from_sets(c.py__getattribute__(*args, **kwargs) for c in self._set)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 430, in from_sets
    for set_ in sets:
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 496, in <genexpr>
    return ValueSet.from_sets(c.py__getattribute__(*args, **kwargs) for c in self._set)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 83, in py__getattribute__
    names = self.goto(name_or_str, name_context, analysis_errors)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 72, in goto
    names = finder.filter_name(filters, name_or_str)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/finder.py", line 35, in filter_name
    for filter in filters:
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/base_value.py", line 62, in _get_value_filters
    yield from self.get_filters(origin_scope=origin_scope)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/value/module.py", line 63, in get_filters
    ParserTreeFilter(
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/filters.py", line 138, in __init__
    super().__init__(parent_context, node_context)
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/filters.py", line 85, in __init__
    module_context = node_context.get_root_context()
  File "/home/colin/workspace/illumon/core.new/jedi-numpy/lib/python3.10/site-packages/jedi/inference/context.py", line 111, in get_root_context
    parent_context = self.parent_context
RecursionError: maximum recursion depth exceeded

Workaround

If you have already imported the numpy module, this will not happen, as long as you don't hit cached results (i.e. import numpy above before the Interpreter is created). This suggests to me at least that this is an edge case in reading types from not-yet-imported modules, rather than a general issue.

On the other hand, a call to jedi.preload_module("numpy") does not resolve this by itself resolve this.

davidhalter commented 3 months ago

It feels like this is definitely a bug, _merge_name_docs should not be called like this I guess. There are other cases where RecursionErrors could be caused by legitimate cases, but this is not inference.

If you want to help, it would really be awesome to have a smaller reproducer for this. This is probably reproducible with a few lines of Python code.

yut23 commented 2 months ago

I got annoyed by my editor lagging out whenever I typed np., so I trimmed it down a simple reproducer: https://github.com/yut23/jedi/commit/99e811ae152e9de7262211bd0942efd2660aadbb

The important part appears to be that the inner stub imports matrix from the outer module with an alias. Changing that line to from repro2020 import matrix makes the RecursionError go away.

niloc132 commented 2 months ago

Thank you for working on this - I managed to break it down so that I could add the bug to numpy 1.26, or remove it from 2.0.x, but was not able to reproduce it in a small, simple project. Your example has the highlights: either replace the matrix as matrix with just matrix in inner/__init__.pyi or change the import in __init__.py to use a wildcard - either change and the bug will go away.

Also, and I'm not sure if this is a bug, but omitting the inner/__init__.pyi file will result in repro2020.matrix being described by jedi as having a "module" type rather than a "class" type.

davidhalter commented 2 months ago

That sounds really strange. But I guess if that reproduces it, I'm very happy you made it work @yut23.

Does the traceback still look the same? With a lot of _merge_name_docs in there?

I'm not surprised that there is a RecursionError, but I'm surprised that that case comes from _merge_name_docs.