pylint crashing with `concurrent.futures.process.BrokenProcessPool`

[tamird]

Bug description

Observing a pylint crash in GitHub Actions on a fairly large project.

Configuration

[tool.pylint]
disable = [
  "broad-exception-caught",
  "broad-exception-raised",
  "c-extension-no-member",
  "cyclic-import",                   # TODO: We should really fix this one or figure out if it is a false positive.
  "duplicate-code",                  # lol
  "empty-docstring",
  "fixme",
  "import-error",
  "inconsistent-return-statements",
  "invalid-name",
  "line-too-long",
  "missing-class-docstring",
  "missing-function-docstring",
  "missing-module-docstring",
  "multiple-statements",
  "no-member",
  "no-name-in-module",
  "not-an-iterable",
  "not-callable",
  "pointless-string-statement",
  "possibly-used-before-assignment",
  "protected-access",
  "redefined-argument-from-local",
  "redefined-builtin",               # TODO: We should clean this up.
  "redefined-outer-name",            # TODO: We should clean this up.
  "too-few-public-methods",
  "too-many-ancestors",
  "too-many-arguments",
  "too-many-boolean-expressions",
  "too-many-branches",
  "too-many-instance-attributes",
  "too-many-lines",
  "too-many-locals",
  "too-many-nested-blocks",
  "too-many-public-methods",
  "too-many-return-statements",
  "too-many-statements",
  "undefined-variable",              # TODO(https://github.com/pylint-dev/pylint/issues/8425): Remove when false positives are gone.

  # TODO: We should fix this.
  "non-ascii-name",

  "ungrouped-imports",
  "unsubscriptable-object",
  "unsupported-membership-test",
  "unused-argument",
  "unused-variable",             # TODO(https://github.com/pylint-dev/pylint/issues/8425): Remove when false positives are gone.
  "unused-wildcard-import",
  "used-before-assignment",      # TODO(https://github.com/pylint-dev/pylint/issues/8425): Remove when false positives are gone.
  "wildcard-import",
  "wrong-import-order",
  "wrong-import-position",
]

Command used

git ls-files -z '*.py' | xargs -0 poetry run pylint -j 0 --output-format=github

Pylint output

Traceback (most recent call last):
  File "/codebuild/output/src3839760031/src/actions-runner/_work/python_monorepo/python_monorepo/.venv/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
             ^^^^^^^^^^^^
  File "/codebuild/output/src3839760031/src/actions-runner/_work/python_monorepo/python_monorepo/.venv/lib/python3.11/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/codebuild/output/src3839760031/src/actions-runner/_work/python_monorepo/python_monorepo/.venv/lib/python3.11/site-packages/pylint/lint/run.py", line 211, in __init__
    linter.check(args)
  File "/codebuild/output/src3839760031/src/actions-runner/_work/python_monorepo/python_monorepo/.venv/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 678, in check
    check_parallel(
  File "/codebuild/output/src3839760031/src/actions-runner/_work/python_monorepo/python_monorepo/.venv/lib/python3.11/site-packages/pylint/lint/parallel.py", line 153, in check_parallel
    for (
  File "/codebuild/output/src3839760031/src/actions-runner/_work/_tool/Python/3.11.9/x64/lib/python3.11/concurrent/futures/process.py", line 620, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/codebuild/output/src3839760031/src/actions-runner/_work/_tool/Python/3.11.9/x64/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/codebuild/output/src3839760031/src/actions-runner/_work/_tool/Python/3.11.9/x64/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/codebuild/output/src3839760031/src/actions-runner/_work/_tool/Python/3.11.9/x64/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/codebuild/output/src3839760031/src/actions-runner/_work/_tool/Python/3.11.9/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Expected behavior

Not crash.

Pylint version

pylint 3.2.3
astroid 3.2.2
Python 3.11.9 (main, Apr 30 2024, 08:22:59) [Clang 15.0.0 (clang-1500.3.9.4)]

OS / Environment

Ubuntu.

Additional dependencies

No response

[tamird]

This started happening after I enabled a bunch of rules, so OOM is possible.

[jacobtylerwalls]

Thanks for the report. I'm not sure there's anything here that we can use to reproduce the issue, unfortunately. Have you tried reducing the number of jobs?

[tamird]

I was able to increase the memory allocated to the build machine and the problem went away.

Closing - may this help someone in the future!