Maximum recursion depth exceeded during analysis of two files

[psass-edfsf]

Steps to reproduce

1. Ceate a.py:

   
   from b import e, f

def c(a, b, c): g = e(a) h = f({"a": a, "b": "c"}, {"a": b, "b": "d"}, {"a": c, "b": "e"}, h=1) [h["c"].sum()]

2. Create `b.py` in the same directory than `a.py`

from datetime import timedelta

import pandas as pd

def e(y: pd.DataFrame): d = (pd.Series(y.index[1:]) - pd.Series(y.index[:-1])).value_counts() d = d.index[d.index > timedelta(minutes=1)].min() return d

def f(t): return t

3. Run pylint on `a.py`

### Current behavior

***** Module a a.py:4:12: W0621: Redefining name 'c' from outer scope (line 4) (redefined-outer-name) Traceback (most recent call last): File "/home/user/.local/bin/pylint", line 8, in sys.exit(run_pylint()) File "/home/user/.local/lib/python3.8/site-packages/pylint/init.py", line 22, in run_pylint PylintRun(sys.argv[1:]) File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/run.py", line 349, in init linter.check(args) File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 862, in check self._check_files( File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 896, in _check_files self._check_file(get_ast, check_astroid_module, name, filepath, modname) File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 922, in _check_file check_astroid_module(ast_node) File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1054, in check_astroid_module retval = self._check_astroid_module( File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1099, in _check_astroid_module walker.walk(ast_node) File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk self.walk(child) File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk self.walk(child) File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk self.walk(child) File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 72, in walk callback(astroid) File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/base.py", line 1990, in visit_assignname self._check_name("variable", node.name, node) File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/base.py", line 2068, in _check_name if match is None and not _should_exempt_from_invalid_name(node): File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/base.py", line 2044, in _should_exempt_from_invalid_name inferred = utils.safe_infer(node) File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/utils.py", line 1143, in safe_infer value = next(infer_gen) File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred yield next(generator) File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped res = next(generator) File "/home/user/.local/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts for inferred in stmt.infer(context=context): File "/home/user/.local/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference yield from islice(iterator, size) File "/home/user/.local/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator for result in generator: File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred yield next(generator) (...) File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred yield next(generator) File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped res = next(generator) File "/home/user/.local/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts for inferred in stmt.infer(context=context): File "/home/user/.local/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference yield from islice(iterator, size) File "/home/user/.local/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator for result in generator: File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred yield next(generator) RecursionError: maximum recursion depth exceeded while calling a Python object

### Expected behavior

***** Module a a.py:4:12: W0621: Redefining name 'c' from outer scope (line 4) (redefined-outer-name) a.py:5:8: E1121: Too many positional arguments for function call (too-many-function-args) a.py:5:8: E1123: Unexpected keyword argument 'h' in function call (unexpected-keyword-arg) a.py:6:4: W0106: Expression "[h['c'].sum()]" is assigned to nothing (expression-not-assigned) a.py:1:0: W0611: Unused e imported from b (unused-import)

### pylint --version output

pylint 2.6.0
astroid 2.4.2
Python 3.8.2 (default, Jul 16 2020, 14:00:26) 
[GCC 9.3.0]

[hippo91]

@Pierre-Sassoulas i cannot reproduce it either.

[psass-edfsf]

I also had difficulty reproducing when using another directory than the original one. I managed to do it by installing the following dependencies:

apipkg==1.5
appdirs==1.4.4
aspy.yaml==1.3.0
astroid==2.4.2
attrs==19.3.0
black==19.10b0
certifi==2020.6.20
cfgv==3.2.0
cftime==1.2.1
chardet==3.0.4
click==7.1.2
colorama==0.4.3
confuse==1.3.0
coverage==5.2.1
cycler==0.10.0
decorator==4.4.2
dicttoxml==1.7.4
distlib==0.3.1
eradicate==1.0
execnet==1.7.1
filelock==3.0.12
flake8==3.8.3
freezegun==0.3.15
identify==1.4.28
idna==2.10
importlib-metadata==1.7.0
iniconfig==1.0.1
ipython-genutils==0.2.0
iso8601==0.1.12
isort==5.4.2
jsonschema==3.2.0
jupyter-core==4.6.3
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
matplotlib==3.3.1
mccabe==0.6.1
more-itertools==8.4.0
mypy==0.782
mypy-extensions==0.4.3
nbformat==5.0.7
netCDF4==1.5.4
nodeenv==1.4.0
numpy==1.19.1
packaging==20.4
pandas==0.25.3
pathspec==0.8.0
Pillow==7.2.0
plotly==3.6.1
pluggy==0.13.1
pre-commit==2.7.1
py==1.9.0
pycodestyle==2.6.0
pyflakes==2.2.0
pylint==2.6.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==6.0.1
pytest-cache==1.0
pytest-cov==2.10.1
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
regex==2020.7.14
requests==2.24.0
retrying==1.3.3
scipy==1.5.2
six==1.15.0
toml==0.10.1
tqdm==4.48.2
traitlets==4.3.3
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.25.10
virtualenv==20.0.31
wcwidth==0.2.5
wrapt==1.12.1
xmltodict==0.12.0
zipp==3.1.0

I wanted to pin down the exact dependencies required without setting them with pip freeze, but the problem started to happen everywhere even after deleting the virtualenv (?!).

[hippo91]

@psass-edfsf in order for us to investigate on this potential issue, could you provide a restricted set of dependencies required to trigger the infinite recursion? Thanks in advance.

[atodorov]

I have also started seeing this in a recent PR on my project which also adds a new checker: https://github.com/kiwitcms/Kiwi/pull/1993 https://github.com/kiwitcms/Kiwi/runs/1247821987?check_suite_focus=true

My specifics are that pylint executes against a large pool of files (site-packages dir) but for some reason this happens only in CI. Locally starts to happen when I have 3x the amount of packages installed.

pylint==2.6.0
pylint-django==2.3.0
pylint-plugin-utils==0.6
astroid==2.4.2

Note: as far as we can tell disabling the new checker being added in the PR above doesn't cause the recursion issues. This is very puzzling. Do you have any hints on how to investigate this ?

[atodorov]

Possible duplicate https://github.com/PyCQA/pylint/issues/3602

[atodorov]

I've been able to dig more information around this. For me it happens when executing pylint against site-packages/setuptools/ like so:

pylint --disable=all /home/senko/.virtualenvs/kiwi_django3/lib/python3.6/site-packages/setuptools/

Attached is the full traceback I get - traceback.txt

[hippo91]

@atodorov thanks for your feed back. We will investigate this ASAP.

[timstaley]

Hi, I've also encountered this and seem to have boiled it down to a smaller reproducible example:

New virtualenv, Python 3.8.0 (Python 3.6 seems unaffected, oddly). pip install pylint pandas

Drop this into mwe.py

import pandas as pd

pd.merge()

Then pylint mwe.py gives

<many repeats>
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
RecursionError: maximum recursion depth exceeded while calling a Python object

pip freeze lists the following

astroid==2.4.2
isort==5.6.4
lazy-object-proxy==1.4.3
mccabe==0.6.1
numpy==1.19.4
pandas==1.1.5
pylint==2.6.0
python-dateutil==2.8.1
pytz==2020.4
six==1.15.0
toml==0.10.2
wrapt==1.12.1

[timstaley]

Workaround; turns out this is a genuinely large recursion problem, and can be gotten around by putting the following in your pylintrc file:

init-hook='import sys; sys.setrecursionlimit(3 * sys.getrecursionlimit())'

(3x works for me, YMMV). h/t https://stackoverflow.com/questions/36496192/pylint-infinite-recursion-in-astriod-package & @exhor :vulcan_salute:

Perhaps there's some way to make the Pylint error more helpful, and point to a relevant FAQ section or something?

[Pierre-Sassoulas]

Maybe we could change the recursion limit directly inside pylint? I encountered this error a lot lately while working with pandas as a dependency. Genuine recursion error would still be caught after 3x the system limit. Does that sound reasonable @hippo91 @AWhetter ?

[AWhetter]

Is it worth trying to figure out why it's recursing so much first? But yes if we think this is good reason for us to need to raise it then let's do that. I think it makes sense to do it only when pylint is run on the command line and not when run as a library, because then the host application should be responsible for it rather than us.

[atodorov]

Is it worth trying to figure out why it's recursing so much first?

Definitely worth trying to figure out IMO.

[hippo91]

@timstaley thanks for your report. This is probably link to #3939 which has been fixed in the current astroids master branch for pandas 1.1.4. However the bug reappears with pandas 1.1.5. It definitely needs investigation so that we can decide whether or not it is legitimate to raise the recursion limit.

[Pierre-Sassoulas]

So it seems that releasing a new version of astroid and pylint would fix the problem for all pandas < 1.1.5. As long as 2.6.1 is not released, the fix to the init_hook provided by @timstaley seems acceptable : https://github.com/PyCQA/pylint/issues/3836#issuecomment-742512541 We can probably wait for the fix for all version of pandas before releasing 2.6.1 ?

[hippo91]

@Pierre-Sassoulas yes we can try to solve the bug for pandas 1.1.5 before releasing what i think could be a major pylint version (2.7).

[Pierre-Sassoulas]

This was solved in pylint 2.7 / astroid 2.5.