search

pylint-dev / pylint

It's not just a linter that annoys you!
https://pylint.readthedocs.io/en/latest/
GNU General Public License v2.0
5.25k stars 1.12k forks source link

RuntimeError: dictionary changed size during iteration #8589

Open xujiboy opened 1 year ago

xujiboy commented 1 year ago

Bug description

When parsing the following file:

import janitor
from metaflow import FlowSpec, step

class LinearFlow(FlowSpec):

    @step
    def start(self):
        self.my_var = 'hello world'
        self.next(self.a)

    @step
    def a(self):
        print('the data artifact is: %s' % self.my_var)
        self.next(self.end)

    @step
    def end(self):
        print('the data artifact is still: %s' % self.my_var)

if __name__ == '__main__':
    LinearFlow()

pylint crashed with a AstroidError and with the following stacktrace:

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 811, in _lint_file check_astroid_module(module) File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1085, in check_astroid_module retval = self._check_astroid_module( File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1135, in _check_astroid_module walker.walk(node) File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 94, in walk self.walk(child) File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 91, in walk callback(astroid) File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/checkers/imports.py", line 534, in visit_import imported_module = self._get_imported_module(node, name) File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/checkers/imports.py", line 883, in _get_imported_module raise astroid.AstroidError from e astroid.exceptions.AstroidError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 775, in _lint_files self._lint_file(fileitem, module, check_astroid_module) File "/home/jixux/miniconda3/envs/metaflow_test/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 813, in _lint_file raise astroid.AstroidError from e astroid.exceptions.AstroidError


### Configuration

_No response_

### Command used

```shell
python linear_flow.py run

Pylint output

linear_flow.py:1:0: F0002: /local/home/jixux/test/metaflow/linear_flow.py: Fatal error while checking '/local/home/jixux/test/metaflow/linear_flow.py'. Please open an issue in our bug tracker so we address this. There is a pre-filled template that you can use in '/local/home/jixux/.cache/pylint/pylint-crash-2023-04-18-16-46-18.txt'. (astroid-error)

Expected behavior

I expect pylint to run without crashing.

Pylint version

pylint 2.17.2
astroid 2.15.3
Python 3.10.0 (default, Mar  3 2022, 09:58:08) [GCC 7.5.0]

OS / Environment

Linux version 5.4.235-151.344.amzn2int.x86_64 (mockbuild@ip-10-0-51-118) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-13) (GCC))

Additional dependencies

aiobotocore==2.4.2 aiohttp==3.8.3 aioitertools==0.11.0 aiosignal==1.3.1 alembic==1.9.2 astroid==2.15.3 asttokens==2.2.1 async-timeout==4.0.2 attrs==22.2.0 autopage==0.5.1 backcall==0.2.0 bigtree==0.6.7 black==22.12.0 boto3==1.24.59 botocore==1.27.59 certifi==2022.12.7 charset-normalizer==2.1.1 click==8.1.3 cliff==4.1.0 cmaes==0.9.1 cmd2==2.4.2 colorlog==6.7.0 contourpy==1.0.6 cycler==0.11.0 Cython==0.29.33 decorator==5.1.1 dill==0.3.6 et-xmlfile==1.1.0 exceptiongroup==1.1.0 executing==1.2.0 fonttools==4.38.0 frozenlist==1.3.3 fsspec==2022.11.0 hdbscan==0.8.29 idna==3.4 importlib-metadata==4.13.0 iniconfig==2.0.0 ipython==8.8.0 isort==5.11.4 jedi==0.18.2 Jinja2==3.1.2 jmespath==1.0.1 joblib==1.2.0 kiwisolver==1.4.4 lazy-object-proxy==1.9.0 lazy_loader==0.1rc2 lightgbm==3.3.3 llvmlite==0.39.1 Mako==1.2.4 markdown-it-py==2.1.0 MarkupSafe==2.1.1 matplotlib==3.6.2 matplotlib-inline==0.1.6 mccabe==0.7.0 mdit-py-plugins==0.3.3 mdurl==0.1.2 metaflow==2.7.18 multidict==6.0.4 multipledispatch==0.6.0 mypy-extensions==0.4.3 natsort==8.2.0 nbclient==0.7.2 networkx==3.0 numba==0.56.4 numpy==1.23.5 nx-altair==0.1.6 openpyxl==3.0.10 optuna==3.0.3 packaging==23.0 pandas==1.5.1 pandas-flavor==0.3.0 parso==0.8.3 pathspec==0.10.3 pbr==5.11.1 pexpect==4.8.0 pickleshare==0.7.5 Pillow==9.4.0 platformdirs==2.6.2 plotly==5.11.0 pluggy==1.0.0 prettytable==3.6.0 prompt-toolkit==3.0.36 ptyprocess==0.7.0 pure-eval==0.2.2 pyarrow==10.0.1 pydot==1.4.2 pyemd==0.5.1 Pygments==2.14.0 pyjanitor==0.24.0 pylint==2.17.2 pynndescent==0.5.8 pyparsing==3.0.9 pyperclip==1.8.2 pytest==7.2.0 python-dateutil==2.8.2 python-dotenv==0.21.0 pytz==2022.6 PyYAML==6.0 requests==2.28.1 s3fs==2022.11.0 s3transfer==0.6.0 scikit-learn==1.2.1 scipy==1.10.1 seaborn==0.12.2 six==1.16.0 SQLAlchemy==1.3.19 stack-data==0.6.2 stevedore==4.1.1 tenacity==8.1.0 threadpoolctl==3.1.0 toml==0.10.2 tomli==2.0.1 tomlkit==0.11.6 toolz==0.12.0 tqdm==4.64.1 traitlets==5.8.1 typing_extensions==4.4.0 umap-learn==0.5.3 urllib3==1.26.13 vega-datasets==0.9.0 watermark==2.3.1 wcwidth==0.2.6 wrapt==1.14.1 xarray==2022.12.0 yarl==1.8.2 zipp==3.11.0

DanielNoord commented 1 year ago

I checked the code and have no idea how the following code can cause that issue:


def _create_dict_items(
    values: Mapping[Any, Any], node: Dict
) -> list[tuple[SuccessfulInferenceResult, SuccessfulInferenceResult]]:
    """Create a list of node pairs to function as the items of a new dict node."""
    elements: list[tuple[SuccessfulInferenceResult, SuccessfulInferenceResult]] = []
    for key, value in values.items():
        key_node = const_factory(key)
        key_node.parent = node
        value_node = const_factory(value)
        value_node.parent = node
        elements.append((key_node, value_node))
    return elements

I might be dumb but the dictionary doesn't seem to change size here.

jacobtylerwalls commented 1 year ago

Thanks for the report, we always like to address crashes when we can reproduce them. Unfortunately, with a fresh venv, and with the pylint/astroid/python versions reported, and with as many of the packages installed as practical (see below), I can't reproduce the crash. Feel free to reopen if you have a reproducer from a fresh venv. Thanks.

pip reports the following package inconsistency trying to reproduce your environment:

ERROR: Cannot install -r requirements.txt (line 34), -r requirements.txt (line 47), -r requirements.txt (line 68) and scipy==1.10.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested scipy==1.10.1
    hdbscan 0.8.29 depends on scipy>=1.0
    lightgbm 3.3.3 depends on scipy
    optuna 3.0.3 depends on scipy<1.9.0 and >=1.7.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
jacobtylerwalls commented 1 year ago

Reported again in #8928.

xujiboy commented 1 year ago

Just want to follow up and FYI that the behavior can be reproduced with this minimal requirement.txt:

metaflow
pyjanitor==0.24.0

However, the error went away if I upgrade pyjanitor to the latest release 0.25.0.

mjimlittle commented 4 months ago

Run into the same issue today as well.

The strange thing is, this is only happening in the CI checks in one of our company's repos. When executing the same thing locally Pylint works as expected.