search

pylint-dev / pylint

It's not just a linter that annoys you!
https://pylint.readthedocs.io/en/latest/
GNU General Public License v2.0
5.23k stars 1.12k forks source link

Crash with AstroidError 'Could not find <FunctionDef.warning_logger' #8049

Open q-wertz opened 1 year ago

q-wertz commented 1 year ago

Bug description

Pylint crashes in a quite complex scenario when testing the whole project. Could not really reproduce it in a minimal example.

The strange thing is, that it seems to have problems with the logging library. When I switch warning(…) to something else it seems to work fine. 

import logging

_log: logging.Logger = logging.getLogger(__name__)

class Test:

    [...]

    def somefunction(self):
        _log.warning("TEST")

When only testing the file that fails, there is no crash...

Configuration

No response

Command used

python3 -m pylint --output-format=colorized myproject

Pylint output

Traceback (most recent call last):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1405, in infer_all
    return list(node.infer(context=context))
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 169, in infer
    yield from self._infer(context=context, **kwargs)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 142, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 111, in wrapped
    for res in _func(node, context, **kwargs):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/inference.py", line 372, in infer_attribute
    yield from owner.igetattr(self.attrname, context)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/bases.py", line 270, in igetattr
    yield from _infer_stmts(
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/bases.py", line 175, in _infer_stmts
    for inf in stmt.infer(context=context):  # type: ignore[union-attr]
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 142, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 111, in wrapped
    for res in _func(node, context, **kwargs):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/bases.py", line 175, in _infer_stmts
    for inf in stmt.infer(context=context):  # type: ignore[union-attr]
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 142, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 111, in wrapped
    for res in _func(node, context, **kwargs):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/inference.py", line 270, in infer_call
    for callee in self.func.infer(context):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 142, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 111, in wrapped
    for res in _func(node, context, **kwargs):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/inference.py", line 358, in infer_attribute
    for owner in self.expr.infer(context):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 142, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/decorators.py", line 111, in wrapped
    for res in _func(node, context, **kwargs):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/inference.py", line 231, in infer_name
    frame, stmts = self.lookup(self.name)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_classes.py", line 367, in lookup
    return self.scope().scope_lookup(self, name)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1766, in scope_lookup
    return super().scope_lookup(node, name, offset)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1211, in scope_lookup
    return frame._scope_lookup(node, name, offset)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 75, in _scope_lookup
    stmts = _filter_stmts(node, self.locals[name], self, offset)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/filter_statements.py", line 194, in _filter_stmts
    if not (optional_assign or nodes.are_exclusive(_stmts[pindex], node)):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_classes.py", line 141, in are_exclusive
    node.locate_child(previous)[1]
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 419, in locate_child
    raise AstroidError(msg % (repr(child), repr(self)))
astroid.exceptions.AstroidError: Could not find <FunctionDef.warning_logger l.464 at 0x7f9d5cd6dc00> in <If l.463 at 0x7f9d69b16b30>'s children

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 790, in _lint_file
    check_astroid_module(module)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1060, in check_astroid_module
    retval = self._check_astroid_module(
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1110, in _check_astroid_module
    walker.walk(node)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 93, in walk
    self.walk(child)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 93, in walk
    self.walk(child)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 93, in walk
    self.walk(child)
  [Previous line repeated 2 more times]
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 90, in walk
    callback(astroid)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/checkers/base/basic_error_checker.py", line 432, in visit_call
    for inferred in infer_all(node.func):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1409, in infer_all
    raise AstroidError from e
astroid.exceptions.AstroidError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 755, in _lint_files
    self._lint_file(fileitem, module, check_astroid_module)
  File "/home/user/VCS/myproject/.venv/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 792, in _lint_file
    raise astroid.AstroidError from e
astroid.exceptions.AstroidError

Expected behavior

Pylint version

pylint 2.15.10
astroid 2.13.2
Python 3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0]

OS / Environment

Manjaro

Additional dependencies

aiohttp==3.8.3 aiosignal==1.3.1 alabaster==0.7.12 altair==4.2.0 appdirs==1.4.4 astroid==2.13.2 asttokens==2.2.1 async-timeout==4.0.2 attrs==22.2.0 Babel==2.11.0 backcall==0.2.0 bcrypt==4.0.1 beautifulsoup4==4.11.1 branca==0.6.0 cartes==0.7.4 Cartopy==0.20.3 certifi==2022.12.7 cffi==1.15.1 charset-normalizer==2.1.1 click==8.1.3 click-plugins==1.1.1 cligj==0.7.2 coloredlogs==15.0.1 comm==0.1.2 commonmark==0.9.1 contourpy==1.0.6 coordinate-magic==0.3.6 coverage==6.5.0 cramjam==2.6.2 cryptography==39.0.0 cycler==0.11.0 debugpy==1.6.5 decorator==5.1.1 dill==0.3.6 docutils==0.17.1 entrypoints==0.4 exceptiongroup==1.1.0 executing==1.2.0 fastparquet==2022.12.0 Fiona==1.8.22 flake8==6.0.0 flake8-codeclimate==0.3.0 flatbuffers==23.1.4 folium==0.13.0 fonttools==4.38.0 frozenlist==1.3.3 fsspec==2022.11.0 geopandas==0.12.2 gitdb==4.0.10 GitPython==3.1.30 haggis==0.8.0 humanfriendly==10.0 idna==3.4 imagesize==1.4.1 iniconfig==2.0.0 ipykernel==6.20.1 ipyleaflet==0.17.2 ipython==8.8.0 ipywidgets==8.0.4 isort==5.11.4 jedi==0.18.2 Jinja2==3.1.2 jsonschema==4.17.3 jupyter_client==7.4.8 jupyter_core==5.1.3 jupyterlab-widgets==3.0.5 kiwisolver==1.4.4 lazy-object-proxy==1.9.0 lxml==4.9.2 MarkupSafe==2.1.1 matplotlib==3.6.2 matplotlib-inline==0.1.6 mccabe==0.7.0 memray==1.5.0 metar==1.9.0 mpmath==1.2.1 multidict==6.0.4 munch==2.5.0 mypy==0.991 mypy-extensions==0.4.3 nest-asyncio==1.5.6 networkx==3.0 numpy==1.24.1 onnxruntime==1.13.1 openap==1.3 packaging==23.0 pandas==1.5.2 pandas-stubs==1.5.2.230105 paramiko==2.12.0 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 Pillow==9.4.0 platformdirs==2.6.2 pluggy==1.0.0 prompt-toolkit==3.0.36 protobuf==4.21.12 psutil==5.9.4 ptyprocess==0.7.0 pure-eval==0.2.2 pycodestyle==2.10.0 pycparser==2.21 pydocstyle==6.2.3 pyflakes==3.0.1 Pygments==2.14.0 pylint==2.15.10 pylint-codeclimate==1.2.4 pylint-exit==1.2.0 pymodes==2.14 PyNaCl==1.5.0 pyOpenSSL==23.0.0 pyparsing==3.0.9 pyproj==3.4.1 pyrsistent==0.19.3 pyshp==2.3.1 pytest==7.2.0 pytest-cov==4.0.0 python-dateutil==2.8.2 pytz==2022.7 PyYAML==6.0 pyzmq==24.0.1 requests==2.28.1 requests-pkcs12==1.14 rich==13.0.1 ruff==0.0.218 scikit-fuzzy==0.4.2 scipy==1.10.0 semantic-version==2.10.0 setuptools-scm==7.1.0 shapely==2.0.0 six==1.16.0 smmap==5.0.0 snakeviz==2.1.1 snowballstemmer==2.2.0 soupsieve==2.3.2.post1 Sphinx==5.3.0 sphinx-rtd-theme==1.1.1 sphinx_autodoc_typehints==1.20.1 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sphinxcontrib.applehelp==1.0.3 stack-data==0.6.2 sympy==1.11.1 tomli==2.0.1 tomlkit==0.11.6 toolz==0.12.0 tornado==6.2 tqdm==4.64.1 traffic==2.8.1 traitlets==5.8.1 traittypes==0.2.1 types-pytz==2022.7.0.0 types-tqdm==4.64.7.9 typing_extensions==4.4.0 urllib3==1.26.13 wcwidth==0.2.5 webcolors==1.12 webcolors-stubs==0.0.3 widgetsnbextension==4.0.5 wrapt==1.14.1 xyzservices==2022.9.0 yapf==0.32.0 yarl==1.8.2

DanielNoord commented 1 year ago

Could you test this with the latest version on the astroid 2.12.x versions? So no 2.13.x?

q-wertz commented 1 year ago

Using

pylint 2.15.10
astroid 2.12.13
Python 3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0]

I get the same behavior…

clavedeluna commented 1 year ago

I'm not able to reproduce this 🤔

q-wertz commented 1 year ago

Some other things that interfere with the default logging behavior (maybe it helps)

  1. warnings from the warnings module are also captured 
    # See https://docs.python.org/3/library/logging.html#logging.captureWarnings
logging.captureWarnings(capture=True)
  2. A trace level is monkey patched to the logging levels using haggis library 
    haggis.logs.add_trace_level()

But all of these things happen at runtime and thus should not interfere with pylint, correct?

jacobtylerwalls commented 1 year ago

It depends on whether they happen at import time. pylint imports modules as part of its analysis.

jacobtylerwalls commented 1 year ago

Notice the function referred to in the traceback comes from the haggis library: https://github.com/madphysicist/haggis/blob/ed2b3ff521c7e5629deaf8cb045cdd95b85920f9/src/haggis/logs.py#L464

q-wertz commented 1 year ago

Notice the function referred to in the traceback comes from the haggis library: https://github.com/madphysicist/haggis/blob/ed2b3ff521c7e5629deaf8cb045cdd95b85920f9/src/haggis/logs.py#L464

Ah thanks. I thought as as in the stack trace there is never a reference to haggis it is something internal and not the one from there… Do you think I have to open an issue there or does this have to be fixed in Astroid/pylint?

jacobtylerwalls commented 1 year ago

Still needs someone to do some investigation: either there is something we can fix in astroid, or there's not (because it's too hard to deal with how monkey-patched this situation is). In the latter case, we could consider catching the AstroidError in pylint, again, if it's easy enough to tell this scenario apart from the generic something-went-wrong-in-astroid in which we don't want to have a wide, lazy catch for everything.

Pierre-Sassoulas commented 1 year ago

Not sure if it's reasonable to handle conditionally defined function in astroid. We have a hard time with control flow already, and this is one step above it. Also how would we differentiate between unexpected astroid error and unexpected plugin error ? Checking the traceback programmatically for "pure astroidness" ?

q-wertz commented 1 year ago

The problem currently is, that it completely crashes pylint (for that file) at that point and the result code quality of the whole project is set to 0.0…

jacobtylerwalls commented 1 year ago

Right, for that reason I think we need to leave this open for a little investigation.

jacobtylerwalls commented 11 months ago

Predates 3.0, suggesting we remove the blocker label.