Closed folkvir closed 3 years ago
I can confirm that we're seeing the same behavior on CI of apache/superset
from time to time after bumping pylint
from 2.9.6
to 2.10.2
(https://github.com/apache/superset/pull/16463)
Can confirm we saw the same thing in open-telemetry/opentelemetry-python-contrib
https://github.com/open-telemetry/opentelemetry-python-contrib/pull/633. Tested with 2.11.1
today and the issue is still present.
@codeboten were you able to reproduce this locally? I tried to run pylint
over your codebase 15+ times in my own environment and didn't get a "hang".
@DanielNoord thanks for the quick reply! I was originally using tox -e lint
to reproduce the issue but managed to narrow it down to a single package in my envionrment. I run the following command from the root of that repo, and then the prompt never returns:
pylint instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry
It looks like it's specifically this file:
instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/instrumentation/flask/__init__.py
I was originally using
tox -e lint
to reproduce the issue but managed to narrow it down to a single package in my envionrment. I run the following command from the root of that repo, and then the prompt never returns:
pylint instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry
@codeboten I was also going to add that hanging on the apache/superset
repo happens both on GitHub Actions and via tox -e pylint
, but I don't believe I was able to repro directly from the CLI (but I probably didn't test that as thoroughly). I can try to repro and see if I can narrow it down to a specific file.
@codeboten I can't seem to reproduce this in a fresh environment. See the following commands:
Last login: Tue Sep 21 19:56:50 on ttys001
❯ gh repo clone open-telemetry/opentelemetry-python-contrib
Cloning into 'opentelemetry-python-contrib'...
remote: Enumerating objects: 21372, done.
remote: Counting objects: 100% (133/133), done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 21372 (delta 39), reused 102 (delta 29), pack-reused 21239
Receiving objects: 100% (21372/21372), 9.00 MiB | 7.32 MiB/s, done.
Resolving deltas: 100% (10888/10888), done.
❯ cd /Users/daniel/DocumentenLaptop/opentelemetry-python-contrib
❯ pyenv virtualenv 3.9.5 opentel
Looking in links: /var/folders/pn/p_xntssj1cgfh8wwwpy8br580000gn/T/tmpz_k3d1rn
Requirement already satisfied: setuptools in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (56.0.0)
Requirement already satisfied: pip in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (21.1.1)
❯ pyenv activate opentel
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
❯ pip install -r /Users/daniel/DocumentenLaptop/opentelemetry-python-contrib/dev-requirements.txt
Collecting pylint<2.10
Using cached pylint-2.9.6-py3-none-any.whl (375 kB)
Collecting flake8~=3.7
Using cached flake8-3.9.2-py2.py3-none-any.whl (73 kB)
Collecting isort~=5.6
Using cached isort-5.9.3-py3-none-any.whl (106 kB)
Collecting black==19.*,>=19.3b0
Using cached black-19.10b0-py36-none-any.whl (97 kB)
Collecting httpretty~=1.0
Using cached httpretty-1.1.4-py3-none-any.whl
Collecting mypy==0.790
Using cached mypy-0.790-py3-none-any.whl (2.4 MB)
Collecting sphinx
Using cached Sphinx-4.2.0-py3-none-any.whl (3.1 MB)
Collecting sphinx-rtd-theme~=0.4
Using cached sphinx_rtd_theme-0.5.2-py2.py3-none-any.whl (9.1 MB)
Collecting sphinx-autodoc-typehints
Using cached sphinx_autodoc_typehints-1.12.0-py3-none-any.whl (9.4 kB)
Collecting pytest!=5.2.3
Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting pytest-cov>=2.8
Using cached pytest_cov-2.12.1-py2.py3-none-any.whl (20 kB)
Collecting readme-renderer~=24.0
Using cached readme_renderer-24.0-py2.py3-none-any.whl (15 kB)
Collecting grpcio-tools==1.29.0
Using cached grpcio-tools-1.29.0.tar.gz (2.0 MB)
Collecting mypy-protobuf>=1.23
Using cached mypy_protobuf-2.10-py3-none-any.whl (15 kB)
Collecting protobuf>=3.13.0
Using cached protobuf-3.18.0-cp39-cp39-macosx_10_9_x86_64.whl (1.0 MB)
Collecting appdirs
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting typed-ast>=1.4.0
Using cached typed_ast-1.4.3-cp39-cp39-macosx_10_9_x86_64.whl (226 kB)
Collecting pathspec<1,>=0.6
Using cached pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting regex
Using cached regex-2021.8.28-cp39-cp39-macosx_10_9_x86_64.whl (285 kB)
Collecting attrs>=18.1.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting toml>=0.9.4
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting click>=6.5
Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting typing-extensions>=3.7.4
Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting mypy-extensions<0.5.0,>=0.4.3
Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting grpcio>=1.29.0
Using cached grpcio-1.40.0-cp39-cp39-macosx_10_10_x86_64.whl (4.0 MB)
Collecting mccabe<0.7,>=0.6
Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting astroid<2.7,>=2.6.5
Using cached astroid-2.6.6-py3-none-any.whl (231 kB)
Collecting pyflakes<2.4.0,>=2.3.0
Using cached pyflakes-2.3.1-py2.py3-none-any.whl (68 kB)
Collecting pycodestyle<2.8.0,>=2.7.0
Using cached pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB)
Collecting docutils<0.17
Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting packaging
Using cached packaging-21.0-py3-none-any.whl (40 kB)
Collecting pluggy<2.0,>=0.12
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting py>=1.8.2
Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting iniconfig
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting coverage>=5.2.1
Using cached coverage-5.5-cp39-cp39-macosx_10_9_x86_64.whl (207 kB)
Collecting bleach>=2.1.0
Using cached bleach-4.1.0-py2.py3-none-any.whl (157 kB)
Collecting Pygments
Using cached Pygments-2.10.0-py3-none-any.whl (1.0 MB)
Collecting six
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting types-protobuf>=3.17.4
Using cached types_protobuf-3.17.4-py3-none-any.whl (50 kB)
Collecting wrapt<1.13,>=1.11
Using cached wrapt-1.12.1-cp39-cp39-macosx_11_0_x86_64.whl
Collecting lazy-object-proxy>=1.4.0
Using cached lazy_object_proxy-1.6.0-cp39-cp39-macosx_10_14_x86_64.whl (21 kB)
Requirement already satisfied: setuptools>=20.0 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from astroid<2.7,>=2.6.5->pylint<2.10->-r /Users/daniel/DocumentenLaptop/opentelemetry-python-contrib/dev-requirements.txt (line 1)) (56.0.0)
Collecting webencodings
Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting types-futures
Using cached types_futures-3.3.0-py3-none-any.whl (4.4 kB)
Collecting sphinxcontrib-qthelp
Using cached sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting snowballstemmer>=1.1
Using cached snowballstemmer-2.1.0-py2.py3-none-any.whl (93 kB)
Collecting sphinxcontrib-serializinghtml>=1.1.5
Using cached sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl (94 kB)
Collecting sphinxcontrib-devhelp
Using cached sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-htmlhelp>=2.0.0
Using cached sphinxcontrib_htmlhelp-2.0.0-py2.py3-none-any.whl (100 kB)
Collecting imagesize
Using cached imagesize-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting babel>=1.3
Using cached Babel-2.9.1-py2.py3-none-any.whl (8.8 MB)
Collecting sphinxcontrib-jsmath
Using cached sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting Jinja2>=2.3
Using cached Jinja2-3.0.1-py3-none-any.whl (133 kB)
Collecting alabaster<0.8,>=0.7
Using cached alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting sphinxcontrib-applehelp
Using cached sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting requests>=2.5.0
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting pytz>=2015.7
Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.0.1-cp39-cp39-macosx_10_9_x86_64.whl (13 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting pyparsing>=2.0.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Using legacy 'setup.py install' for grpcio-tools, since package 'wheel' is not installed.
Installing collected packages: urllib3, pytz, pyparsing, MarkupSafe, idna, charset-normalizer, certifi, wrapt, webencodings, types-futures, toml, sphinxcontrib-serializinghtml, sphinxcontrib-qthelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-applehelp, snowballstemmer, six, requests, Pygments, py, pluggy, packaging, lazy-object-proxy, Jinja2, iniconfig, imagesize, docutils, babel, attrs, alabaster, typing-extensions, types-protobuf, typed-ast, sphinx, regex, pytest, pyflakes, pycodestyle, protobuf, pathspec, mypy-extensions, mccabe, isort, grpcio, coverage, click, bleach, astroid, appdirs, sphinx-rtd-theme, sphinx-autodoc-typehints, readme-renderer, pytest-cov, pylint, mypy-protobuf, mypy, httpretty, grpcio-tools, flake8, black
Running setup.py install for grpcio-tools ... done
Successfully installed Jinja2-3.0.1 MarkupSafe-2.0.1 Pygments-2.10.0 alabaster-0.7.12 appdirs-1.4.4 astroid-2.6.6 attrs-21.2.0 babel-2.9.1 black-19.10b0 bleach-4.1.0 certifi-2021.5.30 charset-normalizer-2.0.6 click-8.0.1 coverage-5.5 docutils-0.16 flake8-3.9.2 grpcio-1.40.0 grpcio-tools-1.29.0 httpretty-1.1.4 idna-3.2 imagesize-1.2.0 iniconfig-1.1.1 isort-5.9.3 lazy-object-proxy-1.6.0 mccabe-0.6.1 mypy-0.790 mypy-extensions-0.4.3 mypy-protobuf-2.10 packaging-21.0 pathspec-0.9.0 pluggy-1.0.0 protobuf-3.18.0 py-1.10.0 pycodestyle-2.7.0 pyflakes-2.3.1 pylint-2.9.6 pyparsing-2.4.7 pytest-6.2.5 pytest-cov-2.12.1 pytz-2021.1 readme-renderer-24.0 regex-2021.8.28 requests-2.26.0 six-1.16.0 snowballstemmer-2.1.0 sphinx-4.2.0 sphinx-autodoc-typehints-1.12.0 sphinx-rtd-theme-0.5.2 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 toml-0.10.2 typed-ast-1.4.3 types-futures-3.3.0 types-protobuf-3.17.4 typing-extensions-3.10.0.2 urllib3-1.26.6 webencodings-0.5.1 wrapt-1.12.1
WARNING: You are using pip version 21.1.1; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/daniel/.pyenv/versions/3.9.5/envs/opentel/bin/python3.9 -m pip install --upgrade pip' command.
❯ pip install pylint==2.11.1
Collecting pylint==2.11.1
Using cached pylint-2.11.1-py3-none-any.whl (392 kB)
Requirement already satisfied: toml>=0.7.1 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from pylint==2.11.1) (0.10.2)
Requirement already satisfied: isort<6,>=4.2.5 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from pylint==2.11.1) (5.9.3)
Collecting astroid<2.9,>=2.8.0
Using cached astroid-2.8.0-py3-none-any.whl (242 kB)
Requirement already satisfied: mccabe<0.7,>=0.6 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from pylint==2.11.1) (0.6.1)
Collecting platformdirs>=2.2.0
Using cached platformdirs-2.3.0-py3-none-any.whl (13 kB)
Requirement already satisfied: typing-extensions>=3.10.0 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from pylint==2.11.1) (3.10.0.2)
Requirement already satisfied: wrapt<1.13,>=1.11 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from astroid<2.9,>=2.8.0->pylint==2.11.1) (1.12.1)
Requirement already satisfied: lazy-object-proxy>=1.4.0 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from astroid<2.9,>=2.8.0->pylint==2.11.1) (1.6.0)
Requirement already satisfied: setuptools>=20.0 in /Users/daniel/.pyenv/versions/3.9.5/envs/opentel/lib/python3.9/site-packages (from astroid<2.9,>=2.8.0->pylint==2.11.1) (56.0.0)
Installing collected packages: platformdirs, astroid, pylint
Attempting uninstall: astroid
Found existing installation: astroid 2.6.6
Uninstalling astroid-2.6.6:
Successfully uninstalled astroid-2.6.6
Attempting uninstall: pylint
Found existing installation: pylint 2.9.6
Uninstalling pylint-2.9.6:
Successfully uninstalled pylint-2.9.6
Successfully installed astroid-2.8.0 platformdirs-2.3.0 pylint-2.11.1
WARNING: You are using pip version 21.1.1; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/daniel/.pyenv/versions/3.9.5/envs/opentel/bin/python3.9 -m pip install --upgrade pip' command.
❯ pylint instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry
************* Module opentelemetry
instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/__init__.py:1:0: F0010: error while code parsing: Unable to load file instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/__init__.py:
[Errno 2] No such file or directory: 'instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/__init__.py' (parse-error)
I do wonder why pylint
is looking for a __init__.py
file. This might be because the top-level directory are modules so it expects the rest to be as well. I know that (namespace
) packages are something that pylint
has troubling dealing with at times. However, I also don't know what changed about this in 2.10
.
Thanks for adding your reproduction steps @DanielNoord. I was able to follow your steps and reproduce pylint not returning by running the following command before running pylint. The eachdist script iterates through the packages in this repo and installs them via pip:
./scripts/eachdist.py install
pylint instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/instrumentation/flask
Interesting sidenote this does not hang:
pylint instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry
but this does:
pylint instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/instrumentation/flask
Thank you! I have actually managed to break pylint
/astroid
myself after running tox
a couple of times (I guess it calls the script you refer to).
Investigating now, although I think this requires quite advanced knowledge of astroid
I'm going to see if I can find what's wrong.
I found a solution! Although I have no idea what will break when I apply it, I do know where we need to look to fully solve this! Going to see if I can make all tests pass with the change, but the culprit is in https://github.com/PyCQA/astroid/commit/cf6528cbc158097c4903f0cab68242ff14bb591b#
@folkvir would you be able to test if installing astroid
from this branch fixes the issues on your repository?
Never mind, pre-commit
doesn't like my change. Going to fix!
@folkvir, @codeboten, @villebro could you try disabling the design-analysis checkers via --disable=too-many-ancestors,too-many-instance-attributes,too-few-public-methods,too-many-public-methods
and see if that affects your performance? See https://github.com/PyCQA/astroid/pull/1188#issuecomment-924707326 for more context.
@nelfin I'll try to carve out time for this later this week 👍 One thing worth noting - we've been getting flip-floppy false positives on too-many-instance-attributes
and useless-suppression
, which was the original reason for bumping to 2.10
in the first place (see https://github.com/apache/superset/pull/16463 for more details). While this is most likely unrelated, I'd be curious to hear if anyone else has seen this behavior (the reason I'm asking is because you mentioned disabling too-many-instance-attributes
above)
I'd be curious to hear if anyone else has seen this behavior
There is this issue I think : https://github.com/PyCQA/pylint/issues/2366 and we also encountered the problem ourselves in astroid
@villebro I have looked at #2366 and submitted a fix, however, I don't think this will fix your issue.
I did notice a pattern within some of the checkers that creates useless-suppression
false positives though.
Would you mind opening an issue with a reproducible example? I'm going to see if I can tackle these useless-suppression
false positives and an issue would help keep track of progress!
Bug description
Configuration
Command used
Pylint output
Expected behavior
should passed
Pylint version
OS / Environment
Centos 7.9 Python 3.8.6
Additional dependencies
amqp==5.0.6 aniso8601==9.0.1 astroid==2.7.3 attrs==21.2.0 billiard==3.6.4.0 black==21.8b0 cached-property==1.5.2 celery==5.1.2 celery-redbeat==2.0.0 certifi==2021.5.30 charset-normalizer==2.0.4 click==7.1.2 click-didyoumean==0.0.3 click-plugins==1.1.1 click-repl==0.2.0 dataclasses==0.8 dnspython==2.1.0 dukpy==0.2.3 fakeredis==1.6.0 Flask==2.0.1 flask-restx==0.5.1 gevent==21.8.0 greenlet==1.1.1 idna==3.2 importlib-metadata==4.8.1 iniconfig==1.1.1 isort==5.9.3 itsdangerous==2.0.1 Jinja2==3.0.1 jsonschema==3.2.0 kamene==0.32 kombu==5.1.0 lazy-object-proxy==1.6.0 ldap3==2.9.1 MarkupSafe==2.0.1 mccabe==0.6.1 mypy-extensions==0.4.3 packaging==21.0 pathspec==0.9.0 platformdirs==2.3.0 pluggy==0.13.1 prompt-toolkit==3.0.20 py==1.10.0 pyasn1==0.4.8 pycurl==7.44.1 pylint==2.10.2 pyodbc==4.0.32 PyPAC==0.15.0 pyparsing==2.4.7 pyrsistent==0.18.0 pytest==6.2.4 pytest-pylint==0.18.0 pytest-sugar==0.9.4 python-dateutil==2.8.2 python-dotenv==0.19.0 pytz==2021.1 PyYAML==5.4.1 redis==3.5.3 regex==2021.8.28 requests==2.26.0 setproctitle==1.2.2 six==1.16.0 sortedcontainers==2.4.0 tenacity==8.0.1 termcolor==1.1.0 tld==0.12.6 toml==0.10.2 tomli==1.2.1 typed-ast==1.4.3 typing-extensions==3.10.0.2 urllib3==1.26.6 uWSGI==2.0.19.1 vine==5.0.0 wcwidth==0.2.5 Werkzeug==2.0.1 wrapt==1.12.1 xxhash==2.0.2 zipp==3.5.0 zope.event==4.5.0 zope.interface==5.4.0 zstandard==0.15.2