"Loop subgraph" panic on main.

[jsirois]

On main @ d39465be54c59af3b2650bdc245431d440795fc0 I consistently see a Loop subgraph panic from here: https://github.com/pantsbuild/pants/blob/d39465be54c59af3b2650bdc245431d440795fc0/src/rust/engine/rule_graph/src/builder.rs#L680-L683

$ RUST_BACKTRACE=1 ./build-support/bin/generate_all_lockfiles.sh --pex
17:11:53.96 [INFO] Completed: Generate lockfile for python-default
17:11:53.97 [INFO] Wrote lockfile for the resolve `python-default` to 3rdparty/python/user_reqs.lock
17:11:53.97 [INFO] Filesystem changed during run: retrying `GenerateLockfilesGoal` in 500ms...
thread '<unnamed>' panicked at 'Loop subgraph: digraph {
    0 [ label = "NodeIndex(81626): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.graph:734:extract_unmatched_build_file_globs(GlobalOptions) -> UnmatchedBuildFileGlobs), in: (()), out: (())))" ]
    1 [ label = "NodeIndex(81629): ParamsLabeled(node: @rule(<intrinsic>(Digest) -> DigestContents), in: (PathGlobs), out: (PathGlobs))" ]
    2 [ label = "NodeIndex(81630): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.build_files:38:evaluate_preludes(BuildFileOptions) -> BuildFilePreludeSymbols, gets=[Get(DigestContents, PathGlobs)]), in: (()), out: (())))" ]
    3 [ label = "NodeIndex(81631): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.build_files:100:parse_address_family(Parser, BuildFileOptions, BuildFilePreludeSymbols, AddressFamilyDir) -> AddressFamily, gets=[Get(DigestContents, PathGlobs)]), in: (AddressFamilyDir), out: (())))" ]
    4 [ label = "NodeIndex(81632): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.build_files:148:find_target_adaptor(Address) -> TargetAdaptor, gets=[Get(AddressFamily, AddressFamilyDir)]), in: (Address), out: (())))" ]
    5 [ label = "NodeIndex(81633): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(<intrinsic>(PathGlobs) -> Paths), in: (PathGlobs), out: (())))" ]
    2 -> 1 [ label = "Get(DigestContents, PathGlobs)" ]
    3 -> 1 [ label = "Get(DigestContents, PathGlobs)" ]
    3 -> 2 [ label = "BuildFilePreludeSymbols" ]
    4 -> 3 [ label = "Get(AddressFamily, AddressFamilyDir)" ]
}
', /home/jsirois/dev/pantsbuild/pants/src/rust/engine/rule_graph/src/builder.rs:680:11
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: rule_graph::builder::Builder<R>::monomorphize
             at ./src/rust/engine/rule_graph/src/builder.rs:680:11
   3: rule_graph::builder::Builder<R>::graph
             at ./src/rust/engine/rule_graph/src/builder.rs:208:41
   4: rule_graph::RuleGraph<R>::new
             at ./src/rust/engine/rule_graph/src/lib.rs:249:5
   5: engine::context::Core::new
             at ./src/rust/engine/src/context.rs:458:22
   6: engine::externs::interface::scheduler_create::{{closure}}
             at ./src/rust/engine/src/externs/interface.rs:648:7
   7: task_executor::Executor::enter
             at ./src/rust/engine/task_executor/src/lib.rs:102:5
   8: engine::externs::interface::scheduler_create
             at ./src/rust/engine/src/externs/interface.rs:645:14
   9: engine::externs::interface::__pyfunction_scheduler_create::{{closure}}
             at ./src/rust/engine/src/externs/interface.rs:613:1
  10: std::panicking::try::do_call
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492:40
  11: std::panicking::try
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456:19
  12: std::panic::catch_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137:14
  13: engine::externs::interface::__pyfunction_scheduler_create
             at ./src/rust/engine/src/externs/interface.rs:613:1
  14: _PyMethodDef_RawFastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:659:19
  15: _PyCFunction_FastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:732:14
  16: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4568:9
  17: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3093:23
  18: PyEval_EvalFrameEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:547:12
  19: _PyEval_EvalCodeWithName
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3930:14
  20: _PyFunction_FastCallDict
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:376:14
  21: _PyObject_Call_Prepend
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:906:14
  22: slot_tp_init
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/typeobject.c:6636:15
  23: type_call
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/typeobject.c:971:19
  24: _PyObject_FastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:199:19
  25: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4619:17
  26: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3139:19
  27: PyEval_EvalFrameEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:547:12
  28: _PyEval_EvalCodeWithName
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3930:14
  29: _PyFunction_FastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:433:12
  30: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  31: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3139:19
  32: PyEval_EvalFrameEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:547:12
  33: _PyEval_EvalCodeWithName
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3930:14
  34: _PyFunction_FastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:433:12
  35: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  36: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3093:23
  37: PyEval_EvalFrameEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:547:12
  38: _PyEval_EvalCodeWithName
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3930:14
  39: _PyFunction_FastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:433:12
  40: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  41: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3093:23
  42: PyEval_EvalFrameEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:547:12
  43: _PyEval_EvalCodeWithName
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3930:14
  44: _PyFunction_FastCallKeywords
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:433:12
  45: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  46: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3139:19
  47: function_code_fastcall
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:283:14
  48: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  49: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3110:23
  50: function_code_fastcall
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:283:14
  51: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  52: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3093:23
  53: function_code_fastcall
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:283:14
  54: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  55: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3093:23
  56: function_code_fastcall
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Objects/call.c:283:14
  57: call_function
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:4616:17
  58: _PyEval_EvalFrameDefault
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3124:19
  59: PyEval_EvalFrameEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:547:12
  60: _PyEval_EvalCodeWithName
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3930:14
  61: PyEval_EvalCodeEx
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:3959:12
  62: PyEval_EvalCode
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/ceval.c:524:12
  63: run_mod
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/pythonrun.c:1037:9
  64: PyRun_FileExFlags
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/pythonrun.c:990:11
  65: PyRun_SimpleFileExFlags
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Python/pythonrun.c:429:13
  66: pymain_run_file
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Modules/main.c:456:11
  67: pymain_run_filename
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Modules/main.c:1646:22
  68: pymain_run_python
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Modules/main.c:2907:9
  69: pymain_main
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Modules/main.c:3068:5
  70: _Py_UnixMain
             at /home/jsirois/.pyenv/sources/3.7.12/Python-3.7.12/Modules/main.c:3103:12
  71: <unknown>
  72: __libc_start_main
  73: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Exception caught: (pyo3_runtime.PanicException)
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 119, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/pants_runner.py", line 100, in run
    env=CompleteEnvironment(self.env), options_bootstrapper=options_bootstrapper
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 156, in create
    cancellation_latch,
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 79, in _init_graph_session
    bootstrap_options, build_config, dynamic_remote_options
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/init/engine_initializer.py", line 199, in setup_graph
    watch_filesystem=bootstrap_options.watch_filesystem,
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/init/engine_initializer.py", line 323, in setup_graph_extended
    watch_filesystem=watch_filesystem,
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/engine/internals/scheduler.py", line 227, in __init__
    exec_stategy_opts,

Exception message: Loop subgraph: digraph {
    0 [ label = "NodeIndex(81626): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.graph:734:extract_unmatched_build_file_globs(GlobalOptions) -> UnmatchedBuildFileGlobs), in: (()), out: (())))" ]
    1 [ label = "NodeIndex(81629): ParamsLabeled(node: @rule(<intrinsic>(Digest) -> DigestContents), in: (PathGlobs), out: (PathGlobs))" ]
    2 [ label = "NodeIndex(81630): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.build_files:38:evaluate_preludes(BuildFileOptions) -> BuildFilePreludeSymbols, gets=[Get(DigestContents, PathGlobs)]), in: (()), out: (())))" ]
    3 [ label = "NodeIndex(81631): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.build_files:100:parse_address_family(Parser, BuildFileOptions, BuildFilePreludeSymbols, AddressFamilyDir) -> AddressFamily, gets=[Get(DigestContents, PathGlobs)]), in: (AddressFamilyDir), out: (())))" ]
    4 [ label = "NodeIndex(81632): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.build_files:148:find_target_adaptor(Address) -> TargetAdaptor, gets=[Get(AddressFamily, AddressFamilyDir)]), in: (Address), out: (())))" ]
    5 [ label = "NodeIndex(81633): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(<intrinsic>(PathGlobs) -> Paths), in: (PathGlobs), out: (())))" ]
    2 -> 1 [ label = "Get(DigestContents, PathGlobs)" ]
    3 -> 1 [ label = "Get(DigestContents, PathGlobs)" ]
    3 -> 2 [ label = "BuildFilePreludeSymbols" ]
    4 -> 3 [ label = "Get(AddressFamily, AddressFamilyDir)" ]
}

NoneType: None
Traceback (most recent call last):
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/309db9d743e5ce409b780562aac4f8562cb8d423/.bootstrap/pex/pex.py", line 517, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/309db9d743e5ce409b780562aac4f8562cb8d423/.bootstrap/pex/pex.py", line 422, in _wrap_coverage
    return runner(*args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/309db9d743e5ce409b780562aac4f8562cb8d423/.bootstrap/pex/pex.py", line 453, in _wrap_profiling
    return runner(*args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/309db9d743e5ce409b780562aac4f8562cb8d423/.bootstrap/pex/pex.py", line 573, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/309db9d743e5ce409b780562aac4f8562cb8d423/.bootstrap/pex/pex.py", line 762, in execute_entry
    return self.execute_module(entry_point, alter_sys)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/309db9d743e5ce409b780562aac4f8562cb8d423/.bootstrap/pex/pex.py", line 774, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/dev/pantsbuild/pants/.pants.d/tmpp1fwzlfm/build-support/bin/_generate_all_lockfiles_helper.py", line 233, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/pants/.pants.d/tmpp1fwzlfm/build-support/bin/_generate_all_lockfiles_helper.py", line 223, in main
    update_default_lockfiles(specified=[Lambdex.options_scope])
  File "/home/jsirois/dev/pantsbuild/pants/.pants.d/tmpp1fwzlfm/build-support/bin/_generate_all_lockfiles_helper.py", line 207, in update_default_lockfiles
    subprocess.run(args, check=True)
  File "/usr/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['./pants', '--concurrent', "--python-interpreter-constraints=['CPython>=3.7,<4']", '--autoflake-version=autoflake==1.4', '--autoflake-extra-requirements=[]', '--autoflake-lockfile=src/python/pants/backend/python/lint/autoflake/autoflake.lock', "--autoflake-interpreter-constraints=['CPython>=3.7,<4']", '--bandit-version=bandit>=1.7.0,<1.8', "--bandit-extra-requirements=['setuptools', 'GitPython==3.1.18']", '--bandit-lockfile=src/python/pants/backend/python/lint/bandit/bandit.lock', "--backend-packages=+['pants.backend.python.lint.bandit']", '--black-version=black==22.1.0', '--black-extra-requirements=[\'typing-extensions>=3.10.0.0; python_version < "3.10"\']', '--black-lockfile=src/python/pants/backend/python/lint/black/black.lock', "--black-interpreter-constraints=['CPython>=3.7,<4']", '--docformatter-version=docformatter>=1.4,<1.5', '--docformatter-extra-requirements=[]', '--docformatter-lockfile=src/python/pants/backend/python/lint/docformatter/docformatter.lock', "--docformatter-interpreter-constraints=['CPython>=3.7,<4']", '--flake8-version=flake8>=3.9.2,<4.0', '--flake8-extra-requirements=[]', '--flake8-lockfile=src/python/pants/backend/python/lint/flake8/flake8.lock', '--flake8-source-plugins=[]', '--isort-version=isort[pyproject,colors]>=5.9.3,<6.0', '--isort-extra-requirements=[]', '--isort-lockfile=src/python/pants/backend/python/lint/isort/isort.lock', "--isort-interpreter-constraints=['CPython>=3.7,<4']", '--pylint-version=pylint>=2.11.0,<2.12', '--pylint-extra-requirements=[]', '--pylint-lockfile=src/python/pants/backend/python/lint/pylint/pylint.lock', '--pylint-source-plugins=[]', "--backend-packages=+['pants.backend.python.lint.pylint']", '--yapf-version=yapf==0.32.0', "--yapf-extra-requirements=['toml']", '--yapf-lockfile=src/python/pants/backend/python/lint/yapf/yapf.lock', "--yapf-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.python.lint.yapf']", '--pyupgrade-version=pyupgrade>=2.31.0,<2.32', '--pyupgrade-extra-requirements=[]', '--pyupgrade-lockfile=src/python/pants/backend/python/lint/pyupgrade/pyupgrade.lock', "--pyupgrade-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.python.lint.pyupgrade']", '--ipython-version=ipython==7.16.1', '--ipython-extra-requirements=[]', '--ipython-lockfile=src/python/pants/backend/python/subsystems/ipython.lock', '--setuptools-version=setuptools>=50.3.0,<58.0', "--setuptools-extra-requirements=['wheel>=0.35.1,<0.38']", '--setuptools-lockfile=src/python/pants/backend/python/subsystems/setuptools.lock', '--setuptools-scm-version=setuptools-scm==6.4.2', '--setuptools-scm-extra-requirements=[]', '--setuptools-scm-lockfile=src/python/pants/backend/python/subsystems/setuptools_scm.lock', "--setuptools-scm-interpreter-constraints=['CPython>=3.7,<4']", '--sphinx-version=sphinx>=4.5.0,<4.6', "--sphinx-extra-requirements=['setuptools']", '--sphinx-lockfile=src/python/pants/backend/python/docs/sphinx/sphinx.lock', "--sphinx-interpreter-constraints=['CPython>=3.7,<4']", '--mypy-version=mypy==0.950', '--mypy-extra-requirements=[]', '--mypy-lockfile=src/python/pants/backend/python/typecheck/mypy/mypy.lock', "--mypy-interpreter-constraints=['CPython>=3.7,<4']", '--mypy-source-plugins=[]', '--mypy-protobuf-version=mypy-protobuf==2.10', '--mypy-protobuf-extra-requirements=[]', '--mypy-protobuf-lockfile=src/python/pants/backend/codegen/protobuf/python/mypy_protobuf.lock', "--mypy-protobuf-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.codegen.protobuf.python']", '--lambdex-version=lambdex==0.1.6', '--lambdex-extra-requirements=[]', '--lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock', "--lambdex-interpreter-constraints=['CPython>=3.7,<3.10']", "--backend-packages=+['pants.backend.awslambda.python']", '--pytest-version=pytest==7.0.1', "--pytest-extra-requirements=['pytest-cov>=2.12,!=2.12.1,<3.1']", '--pytest-lockfile=src/python/pants/backend/python/subsystems/pytest.lock', '--coverage-py-version=coverage[toml]>=5.5,<5.6', '--coverage-py-extra-requirements=[]', '--coverage-py-lockfile=src/python/pants/backend/python/subsystems/coverage_py.lock', "--coverage-py-interpreter-constraints=['CPython>=3.7,<4']", '--terraform-hcl2-parser-version=python-hcl2==3.0.5', '--terraform-hcl2-parser-extra-requirements=[]', '--terraform-hcl2-parser-lockfile=src/python/pants/backend/terraform/hcl2.lock', "--terraform-hcl2-parser-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.terraform']", '--dockerfile-parser-version=dockerfile==3.2.0', '--dockerfile-parser-extra-requirements=[]', '--dockerfile-parser-lockfile=src/python/pants/backend/docker/subsystems/dockerfile.lock', "--dockerfile-parser-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.docker']", '--twine-version=twine>=3.7.1,<3.8', "--twine-extra-requirements=['colorama>=0.4.3']", '--twine-lockfile=src/python/pants/backend/python/subsystems/twine.lock', "--twine-interpreter-constraints=['CPython>=3.7,<4']", '--clang-format-version=clang-format==14.0.3', '--clang-format-extra-requirements=[]', '--clang-format-lockfile=src/python/pants/backend/cc/lint/clangformat/clangformat.lock', "--clang-format-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.cc.lint.clangformat']", '--junit-version=5.7.2', "--junit-artifacts=('org.junit.platform:junit-platform-console:1.7.2', 'org.junit.jupiter:junit-jupiter-engine:{version}', 'org.junit.vintage:junit-vintage-engine:{version}')", '--junit-lockfile=src/python/pants/jvm/test/junit.default.lockfile.txt', '--google-java-format-version=1.13.0', "--google-java-format-artifacts=('com.google.googlejavaformat:google-java-format:{version}',)", '--google-java-format-lockfile=src/python/pants/backend/java/lint/google_java_format/google_java_format.default.lockfile.txt', '--scalafmt-version=3.2.1', "--scalafmt-artifacts=('org.scalameta:scalafmt-cli_2.13:{version}',)", '--scalafmt-lockfile=src/python/pants/backend/scala/lint/scalafmt/scalafmt.default.lockfile.txt', '--scalapb-version=0.11.6', "--scalapb-artifacts=('com.thesamet.scalapb:scalapbc_2.13:{version}',)", '--scalapb-lockfile=src/python/pants/backend/codegen/protobuf/scala/scalapbc.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.protobuf.scala']", '--scalatest-version=3.2.10', "--scalatest-artifacts=('org.scalatest:scalatest_2.13:{version}',)", '--scalatest-lockfile=src/python/pants/backend/scala/subsystems/scalatest.default.lockfile.txt', '--scrooge-version=21.12.0', "--scrooge-artifacts=('com.twitter:scrooge-generator_2.13:{version}',)", '--scrooge-lockfile=src/python/pants/backend/codegen/thrift/scrooge/scrooge.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.thrift.scrooge.scala']", '--java-avro-version=1.11.0', "--java-avro-artifacts=('org.apache.avro:avro-tools:{version}',)", '--java-avro-lockfile=src/python/pants/backend/codegen/avro/java/avro-tools.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.avro.java']", '--ktlint-version=0.45.2', "--ktlint-artifacts=('com.pinterest:ktlint:{version}',)", '--ktlint-lockfile=src/python/pants/backend/kotlin/lint/ktlint/ktlint.lock', "--backend-packages=+['pants.backend.experimental.kotlin.lint.ktlint']", 'generate-lockfiles', "--resolve=['lambdex']"]' returned non-zero exit status 1.

[jsirois]

This bisects to 7eea60ed1378dab1de01787facfd363d35dc672b:

$ git bisect bad
7eea60ed1378dab1de01787facfd363d35dc672b is the first bad commit
commit 7eea60ed1378dab1de01787facfd363d35dc672b
Author: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
Date:   Tue May 24 11:37:37 2022 -0500

    Refactor `Specs` as prework for ignore Specs (#15620)

    Prework for https://github.com/pantsbuild/pants/pull/15619. This does not actually wire up ignore specs yet, but it changes the `specs.py` modeling so that it's trivial to finish the feature.

    As described at https://github.com/pantsbuild/pants/issues/15539, the semantics for ignore specs are simple: ignores always win out over includes.

    Those semantics make modeling this problem easy. What used to be called `Specs` is now called `RawSpecs`: how to find all targets matching the specs, not considering ignores vs includes.

    Then, we have a `Specs` object:

    ```python
    @dataclass(frozen=True)
    class Specs:
        includes: RawSpecs = RawSpecs()
        ignores: RawSpecs = RawSpecs()

And calculating the final `Addresses` is nothing more than:

```python
@rule
async def resolve_addresses_from_specs(specs: Specs) -> Addresses:
    includes, ignores = await MultiGet(
        Get(Addresses, RawSpecs, specs.includes),
        Get(Addresses, RawSpecs, specs.ignores),
    )
    return Addresses(FrozenOrderedSet(includes) - FrozenOrderedSet(ignores))
```

When dealing directly with user specs, i.e. in goals, you should use `Specs` so ignores are included. Otherwise, rules can use either `Specs` or `RawSpecs` for things like `tailor` and dependency inference.

src/python/pants/backend/go/goals/tailor.py | 4 +- src/python/pants/backend/go/target_type_rules.py | 4 +- src/python/pants/backend/go/util_rules/go_mod.py | 4 +- src/python/pants/backend/project_info/peek_test.py | 6 +- .../pants/backend/python/goals/export_test.py | 6 +- src/python/pants/backend/python/goals/setup_py.py | 4 +- src/python/pants/backend/python/goals/tailor.py | 4 +- .../backend/terraform/dependency_inference.py | 4 +- src/python/pants/base/specs.py | 104 ++++++++++++------ src/python/pants/base/specs_parser.py | 63 ++++++++--- src/python/pants/base/specs_parser_test.py | 9 +- src/python/pants/base/specs_test.py | 20 ++-- src/python/pants/bin/local_pants_runner.py | 6 +- src/python/pants/bsp/util_rules/targets.py | 12 ++- src/python/pants/core/goals/tailor.py | 8 +- src/python/pants/core/goals/tailor_test.py | 16 +-- src/python/pants/engine/internals/graph.py | 16 +-- src/python/pants/engine/internals/specs_rules.py | 57 ++++++---- .../pants/engine/internals/specs_rules_test.py | 120 ++++++++++++--------- src/python/pants/engine/target.py | 2 +- src/python/pants/init/specs_calculator.py | 19 ++-- .../jvm/resolve/coursier_fetch_integration_test.py | 6 +- 22 files changed, 306 insertions(+), 188 deletions(-)

[stuhood]

A (shlex-mangled) repro command looks like:

./pants --concurrent '--python-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --autoflake-version=autoflake==1.4 '--autoflake-extra-requirements=[]' --autoflake-lockfile=src/python/pants/backend/python/lint/autoflake/autoflake.lock '--autoflake-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--bandit-version=bandit>=1.7.0,<1.8' '--bandit-extra-requirements=['"'"'setuptools'"'"', '"'"'GitPython==3.1.18'"'"']' --bandit-lockfile=src/python/pants/backend/python/lint/bandit/bandit.lock '--backend-packages=+['"'"'pants.backend.python.lint.bandit'"'"']' --black-version=black==22.1.0 '--black-extra-requirements=['"'"'typing-extensions>=3.10.0.0; python_version < "3.10"'"'"']' --black-lockfile=src/python/pants/backend/python/lint/black/black.lock '--black-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--docformatter-version=docformatter>=1.4,<1.5' '--docformatter-extra-requirements=[]' --docformatter-lockfile=src/python/pants/backend/python/lint/docformatter/docformatter.lock '--docformatter-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--flake8-version=flake8>=3.9.2,<4.0' '--flake8-extra-requirements=[]' --flake8-lockfile=src/python/pants/backend/python/lint/flake8/flake8.lock '--flake8-source-plugins=[]' '--isort-version=isort[pyproject,colors]>=5.9.3,<6.0' '--isort-extra-requirements=[]' --isort-lockfile=src/python/pants/backend/python/lint/isort/isort.lock '--isort-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--pylint-version=pylint>=2.11.0,<2.12' '--pylint-extra-requirements=[]' --pylint-lockfile=src/python/pants/backend/python/lint/pylint/pylint.lock '--pylint-source-plugins=[]' '--backend-packages=+['"'"'pants.backend.python.lint.pylint'"'"']' --yapf-version=yapf==0.32.0 '--yapf-extra-requirements=['"'"'toml'"'"']' --yapf-lockfile=src/python/pants/backend/python/lint/yapf/yapf.lock '--yapf-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.python.lint.yapf'"'"']' '--pyupgrade-version=pyupgrade>=2.31.0,<2.32' '--pyupgrade-extra-requirements=[]' --pyupgrade-lockfile=src/python/pants/backend/python/lint/pyupgrade/pyupgrade.lock '--pyupgrade-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.experimental.python.lint.pyupgrade'"'"']' --ipython-version=ipython==7.16.1 '--ipython-extra-requirements=[]' --ipython-lockfile=src/python/pants/backend/python/subsystems/ipython.lock '--setuptools-version=setuptools>=50.3.0,<58.0' '--setuptools-extra-requirements=['"'"'wheel>=0.35.1,<0.38'"'"']' --setuptools-lockfile=src/python/pants/backend/python/subsystems/setuptools.lock --setuptools-scm-version=setuptools-scm==6.4.2 '--setuptools-scm-extra-requirements=[]' --setuptools-scm-lockfile=src/python/pants/backend/python/subsystems/setuptools_scm.lock '--setuptools-scm-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--sphinx-version=sphinx>=4.5.0,<4.6' '--sphinx-extra-requirements=['"'"'setuptools'"'"']' --sphinx-lockfile=src/python/pants/backend/python/docs/sphinx/sphinx.lock '--sphinx-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --mypy-version=mypy==0.950 '--mypy-extra-requirements=[]' --mypy-lockfile=src/python/pants/backend/python/typecheck/mypy/mypy.lock '--mypy-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--mypy-source-plugins=[]' --mypy-protobuf-version=mypy-protobuf==2.10 '--mypy-protobuf-extra-requirements=[]' --mypy-protobuf-lockfile=src/python/pants/backend/codegen/protobuf/python/mypy_protobuf.lock '--mypy-protobuf-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.codegen.protobuf.python'"'"']' --lambdex-version=lambdex==0.1.6 '--lambdex-extra-requirements=[]' --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock '--lambdex-interpreter-constraints=['"'"'CPython>=3.7,<3.10'"'"']' '--backend-packages=+['"'"'pants.backend.awslambda.python'"'"']' --pytest-version=pytest==7.0.1 '--pytest-extra-requirements=['"'"'pytest-cov>=2.12,!=2.12.1,<3.1'"'"']' --pytest-lockfile=src/python/pants/backend/python/subsystems/pytest.lock '--coverage-py-version=coverage[toml]>=5.5,<5.6' '--coverage-py-extra-requirements=[]' --coverage-py-lockfile=src/python/pants/backend/python/subsystems/coverage_py.lock '--coverage-py-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --terraform-hcl2-parser-version=python-hcl2==3.0.5 '--terraform-hcl2-parser-extra-requirements=[]' --terraform-hcl2-parser-lockfile=src/python/pants/backend/terraform/hcl2.lock '--terraform-hcl2-parser-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.experimental.terraform'"'"']' --dockerfile-parser-version=dockerfile==3.2.0 '--dockerfile-parser-extra-requirements=[]' --dockerfile-parser-lockfile=src/python/pants/backend/docker/subsystems/dockerfile.lock '--dockerfile-parser-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.docker'"'"']' '--twine-version=twine>=3.7.1,<3.8' '--twine-extra-requirements=['"'"'colorama>=0.4.3'"'"']' --twine-lockfile=src/python/pants/backend/python/subsystems/twine.lock '--twine-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --clang-format-version=clang-format==14.0.3 '--clang-format-extra-requirements=[]' --clang-format-lockfile=src/python/pants/backend/cc/lint/clangformat/clangformat.lock '--clang-format-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.experimental.cc.lint.clangformat'"'"']' --junit-version=5.7.2 '--junit-artifacts=('"'"'org.junit.platform:junit-platform-console:1.7.2'"'"', '"'"'org.junit.jupiter:junit-jupiter-engine:{version}'"'"', '"'"'org.junit.vintage:junit-vintage-engine:{version}'"'"')' --junit-lockfile=src/python/pants/jvm/test/junit.default.lockfile.txt --google-java-format-version=1.13.0 '--google-java-format-artifacts=('"'"'com.google.googlejavaformat:google-java-format:{version}'"'"',)' --google-java-format-lockfile=src/python/pants/backend/java/lint/google_java_format/google_java_format.default.lockfile.txt --scalafmt-version=3.2.1 '--scalafmt-artifacts=('"'"'org.scalameta:scalafmt-cli_2.13:{version}'"'"',)' --scalafmt-lockfile=src/python/pants/backend/scala/lint/scalafmt/scalafmt.default.lockfile.txt --scalapb-version=0.11.6 '--scalapb-artifacts=('"'"'com.thesamet.scalapb:scalapbc_2.13:{version}'"'"',)' --scalapb-lockfile=src/python/pants/backend/codegen/protobuf/scala/scalapbc.default.lockfile.txt '--backend-packages=+['"'"'pants.backend.experimental.codegen.protobuf.scala'"'"']' --scalatest-version=3.2.10 '--scalatest-artifacts=('"'"'org.scalatest:scalatest_2.13:{version}'"'"',)' --scalatest-lockfile=src/python/pants/backend/scala/subsystems/scalatest.default.lockfile.txt --scrooge-version=21.12.0 '--scrooge-artifacts=('"'"'com.twitter:scrooge-generator_2.13:{version}'"'"',)' --scrooge-lockfile=src/python/pants/backend/codegen/thrift/scrooge/scrooge.default.lockfile.txt '--backend-packages=+['"'"'pants.backend.experimental.codegen.thrift.scrooge.scala'"'"']' --java-avro-version=1.11.0 '--java-avro-artifacts=('"'"'org.apache.avro:avro-tools:{version}'"'"',)' --java-avro-lockfile=src/python/pants/backend/codegen/avro/java/avro-tools.default.lockfile.txt '--backend-packages=+['"'"'pants.backend.experimental.codegen.avro.java'"'"']' --ktlint-version=0.45.2 '--ktlint-artifacts=('"'"'com.pinterest:ktlint:{version}'"'"',)' --ktlint-lockfile=src/python/pants/backend/kotlin/lint/ktlint/ktlint.lock '--backend-packages=+['"'"'pants.backend.experimental.kotlin.lint.ktlint'"'"']' generate-lockfiles '--resolve=['"'"'lambdex'"'"']'

[jsirois]

This is the minimal repro I find:

./pants --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock generate-lockfiles --resolve=lambdex

Remove any single backend (except 'pants.backend.awslambda.python' which is required for this), and things are green.

[jsirois]

Ok, with this diff:

$ git diff src/rust/
diff --git a/src/rust/engine/rule_graph/src/builder.rs b/src/rust/engine/rule_graph/src/builder.rs
index e60873dfb..535e897ed 100644
--- a/src/rust/engine/rule_graph/src/builder.rs
+++ b/src/rust/engine/rule_graph/src/builder.rs
@@ -5,6 +5,7 @@ use crate::rules::{DependencyKey, ParamTypes, Query, Rule};
 use crate::{params_str, Entry, EntryWithDeps, InnerEntry, RootEntry, RuleEdges, RuleGraph};

 use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
+use std::env;

 use indexmap::IndexSet;
 use internment::Intern;
@@ -478,6 +479,7 @@ impl<R: Rule> Builder<R> {
     let mut iteration = 0;
     let mut maybe_in_loop = HashSet::new();
     let mut looping = false;
+    let looping_threshold = env::var("PANTS_RG_LOOP_THRESH").map_or(100000, |v| v.parse::<usize>().unwrap());
     while let Some(node_id) = to_visit.pop() {
       let node = if let Some(node) = graph[node_id].inner() {
         node
@@ -500,7 +502,7 @@ impl<R: Rule> Builder<R> {
       }

       iteration += 1;
-      if iteration > 100000 {
+      if iteration > looping_threshold {
         looping = true;
       }
       if iteration % 1000 == 0 {
@@ -665,7 +667,7 @@ impl<R: Rule> Builder<R> {
         log::trace!("{}", trace_str);

         maybe_in_loop.insert(node_id);
-        if maybe_in_loop.len() > 5 {
+        if maybe_in_loop.len() > env::var("PANTS_RG_LOOP_MAX").map_or(5, |v| v.parse::<usize>().unwrap()) {
           let subgraph = graph.filter_map(
             |node_id, node| {
               if maybe_in_loop.contains(&node_id) {
@@ -678,7 +680,8 @@ impl<R: Rule> Builder<R> {
           );

           panic!(
-            "Loop subgraph: {}",
+            "Loop subgraph [{} in loop]: {}",
+            maybe_in_loop.len(),
             petgraph::dot::Dot::with_config(&subgraph, &[])
           );
         }

I find non-deterministic success bumping up loop thresholds; so I'm not really sure if the bumps are effective or they just expose randomness better as they get bigger?:

Good:

$ PANTS_RG_LOOP_MAX=6 PANTS_RG_LOOP_THRESH=1000000000000000 ./pants --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock generate-lockfiles --resolve=lambdex
19:40:13.87 [INFO] Initializing scheduler...
19:40:14.23 [INFO] Scheduler initialized.
19:40:18.31 [INFO] Completed: Generate lockfile for lambdex
19:40:18.31 [INFO] Wrote lockfile for the resolve `lambdex` to src/python/pants/backend/python/subsystems/lambdex.lock

Then pkill pantsd and make sure all pantsd are down. Then:

$ PANTS_RG_LOOP_MAX=6 PANTS_RG_LOOP_THRESH=1000000000000000 ./pants --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock generate-lockfiles --resolve=lambdex
19:40:25.91 [INFO] Initializing scheduler...
19:40:26.25 [ERROR] panic at 'Loop subgraph [6 in loop]: digraph {
    0 [ label = "NodeIndex(49419): ParamsLabeled(node: @rule(pants.backend.scala.dependency_inference.rules:165:inject_scala_plugin_dependencies(InjectScalaPluginDependenciesRequest) -> InjectedDependencies, gets=[Get(WrappedTarget, Address), Get(ScalaPluginTargetsForTarget, ScalaPluginsForTargetWithoutResolveRequest)]), in: (InjectScalaPluginDependenciesRequest), out: (InjectScalaPluginDependenciesRequest))" ]
    1 [ label = "NodeIndex(79211): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.options_parsing:38:parse_options(BuildConfiguration, SessionValues) -> _Options), in: (()), out: (())))" ]
    2 [ label = "NodeIndex(79212): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.options_parsing:46:scope_options(Scope, _Options) -> ScopedOptions), in: (Scope), out: (())))" ]
    3 [ label = "NodeIndex(79213): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.option.global_options:1446:construct_scope_() -> GlobalOptions, gets=[Get(ScopedOptions, Scope)]), in: (()), out: (())))" ]
    4 [ label = "NodeIndex(79220): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.graph:108:target_types_to_generate_targets_requests(UnionMembership) -> TargetTypesToGenerateTargetsRequests), in: (()), out: (())))" ]
    5 [ label = "NodeIndex(79222): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.graph:734:extract_unmatched_build_file_globs(GlobalOptions) -> UnmatchedBuildFileGlobs), in: (()), out: (())))" ]
    2 -> 1 [ label = "_Options" ]
    3 -> 2 [ label = "Get(ScopedOptions, Scope)" ]
    5 -> 3 [ label = "GlobalOptions" ]
}
', /home/jsirois/dev/pantsbuild/jsirois-pants/src/rust/engine/rule_graph/src/builder.rs:682
19:40:26.25 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
^CTraceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 119, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/remote_pants_runner.py", line 117, in run
    return self._connect_and_execute(pantsd_handle, start_time)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/remote_pants_runner.py", line 151, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (https://www.pantsbuild.org/docs/getting-help).

[stuhood]

I find non-deterministic success bumping up loop thresholds; so I'm not really sure if the bumps are effective or they just expose randomness better as they get bigger?:

I see deterministic success with a higher "looping" threshold... I think that you might have been running into the fact that we prune all env vars in pantsd?

With your patch applied and --no-pantsd, this command reliably succeeds:

PANTS_RG_LOOP_THRESH=1000000 ./pants --no-pantsd --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" help

---

Given the env var filtering issue, and since I'm hoping to be able to resume #11269 this summer, I'll likely bump the hardcoded loop threshold for now.

[jsirois]

Can't be. Are they pruned for the initial pantsd bring up? As noted I killed pantsd between all attempts.

[stuhood]

Are they pruned for the initial pantsd bring up?

Yes, because pantsd is always forked, and truncates its own env vars as it starts up: https://github.com/pantsbuild/pants/blob/b251df7105ff597031196ab0c85800905727c59a/src/python/pants/pantsd/pants_daemon.py#L176-L181

[jsirois]

So my success vs failure was non-deterministic using existing thresholds. Is that expected? That the algorithm is non-deterministic?

[stuhood]

I'm not sure. But I suspect that one of your successful attempts was using an outdated instance of pantsd where the value was hardcoded, perhaps (assuming that you went through a few iterations before arriving at that patch)...?

Is that expected? That the algorithm is non-deterministic?

It is expected in that we are using Rust's default randomized hash function... but I've never known it to affect success or failure. #11269 switches to deterministic hashing for that and other reasons.

[jsirois]

On main after your fix ./build-support/bin/generate_all_lockfiles.sh --pex is green.

But, with ~my diff re-applied:

$ git diff src/rust/
diff --git a/src/rust/engine/rule_graph/src/builder.rs b/src/rust/engine/rule_graph/src/builder.rs
index 4c9ef3141..2de045e36 100644
--- a/src/rust/engine/rule_graph/src/builder.rs
+++ b/src/rust/engine/rule_graph/src/builder.rs
@@ -5,6 +5,7 @@ use crate::rules::{DependencyKey, ParamTypes, Query, Rule};
 use crate::{params_str, Entry, EntryWithDeps, InnerEntry, RootEntry, RuleEdges, RuleGraph};

 use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
+use std::env;

 use indexmap::IndexSet;
 use internment::Intern;
@@ -478,6 +479,9 @@ impl<R: Rule> Builder<R> {
     let mut iteration = 0;
     let mut maybe_in_loop = HashSet::new();
     let mut looping = false;
+    let loop_thresh =
+      env::var("PANTS_RG_LOOP_THRESH").map_or(10000000, |v| v.parse::<usize>().unwrap());
+    let loop_max = env::var("PANTS_RG_LOOP_MAX").map_or(5, |v| v.parse::<usize>().unwrap());
     while let Some(node_id) = to_visit.pop() {
       let node = if let Some(node) = graph[node_id].inner() {
         node
@@ -507,7 +511,7 @@ impl<R: Rule> Builder<R> {
       // See https://github.com/pantsbuild/pants/issues/11269 for plans to improve this
       // implementation.
       iteration += 1;
-      if iteration > 10000000 {
+      if iteration > loop_thresh {
         looping = true;
       }
       if iteration % 1000 == 0 {
@@ -672,7 +676,7 @@ impl<R: Rule> Builder<R> {
         log::trace!("{}", trace_str);

         maybe_in_loop.insert(node_id);
-        if maybe_in_loop.len() > 5 {
+        if maybe_in_loop.len() > loop_max {
           let subgraph = graph.filter_map(
             |node_id, node| {
               if maybe_in_loop.contains(&node_id) {
@@ -685,7 +689,9 @@ impl<R: Rule> Builder<R> {
           );

           panic!(
-            "Loop subgraph: {}",
+            "Loop subgraph [thresh={}, max={}]: {}",
+            loop_thresh,
+            loop_max,
             petgraph::dot::Dot::with_config(&subgraph, &[])
           );
         }

I get:

$ pkill pantsd
$ pgrep pantsd
$ PANTS_RG_LOOP_MAX=5 PANTS_RG_LOOP_THRESH=100000 ./build-support/bin/generate_all_lockfiles.sh --pex
16:44:37.90 [INFO] Initializing scheduler...
16:44:38.25 [INFO] Scheduler initialized.
16:45:05.80 [INFO] Completed: Generate lockfile for python-default
16:45:05.81 [INFO] Wrote lockfile for the resolve `python-default` to 3rdparty/python/user_reqs.lock
16:45:08.07 [ERROR] panic at 'Loop subgraph [thresh=100000, max=5]: digraph {
    0 [ label = "NodeIndex(23432): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.core.util_rules.archive:98:convert_digest_to_MaybeExtractArchiveRequest(Digest) -> MaybeExtractArchiveRequest), in: (Digest), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest))))" ]
    1 [ label = "NodeIndex(23433): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(Digest) -> Snapshot), in: (Digest), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest))))" ]
    2 [ label = "NodeIndex(23439): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(RemovePrefix) -> Digest), in: (RemovePrefix), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest, RemovePrefix))))" ]
    3 [ label = "NodeIndex(23446): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(MergeDigests) -> Digest), in: (MergeDigests), out: ((Digest, ExportPythonTool, MergeDigests, OptionalPexRequest, PexRequest))))" ]
    4 [ label = "NodeIndex(23451): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(CreateDigest) -> Digest), in: (CreateDigest), out: ((CreateDigest, Digest, ExportPythonTool, OptionalPexRequest, PexRequest))))" ]
    5 [ label = "NodeIndex(23568): ParamsLabeled(node: @rule(<intrinsic>(Process) -> FallibleProcessResult), in: (Process), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest, Process, UnzipBinaryRequest)))" ]
}
', /home/jsirois/dev/pantsbuild/pants/src/rust/engine/rule_graph/src/builder.rs:691
16:45:08.07 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
Exception caught: (pyo3_runtime.PanicException)
...
subprocess.CalledProcessError: Command '['./pants', '--concurrent', "--python-interpreter-constraints=['CPython>=3.7,<4']", '--autoflake-version=autoflake==1.4', '--autoflake-extra-requirements=[]', '--autoflake-lockfile=src/python/pants/backend/python/lint/autoflake/autoflake.lock', "--autoflake-interpreter-constraints=['CPython>=3.7,<4']", '--bandit-version=bandit>=1.7.0,<1.8', "--bandit-extra-requirements=['setuptools', 'GitPython==3.1.18']", '--bandit-lockfile=src/python/pants/backend/python/lint/bandit/bandit.lock', "--backend-packages=+['pants.backend.python.lint.bandit']", '--black-version=black==22.1.0', '--black-extra-requirements=[\'typing-extensions>=3.10.0.0; python_version < "3.10"\']', '--black-lockfile=src/python/pants/backend/python/lint/black/black.lock', "--black-interpreter-constraints=['CPython>=3.7,<4']", '--docformatter-version=docformatter>=1.4,<1.5', '--docformatter-extra-requirements=[]', '--docformatter-lockfile=src/python/pants/backend/python/lint/docformatter/docformatter.lock', "--docformatter-interpreter-constraints=['CPython>=3.7,<4']", '--flake8-version=flake8>=3.9.2,<4.0', '--flake8-extra-requirements=[]', '--flake8-lockfile=src/python/pants/backend/python/lint/flake8/flake8.lock', '--flake8-source-plugins=[]', '--isort-version=isort[pyproject,colors]>=5.9.3,<6.0', '--isort-extra-requirements=[]', '--isort-lockfile=src/python/pants/backend/python/lint/isort/isort.lock', "--isort-interpreter-constraints=['CPython>=3.7,<4']", '--pylint-version=pylint>=2.11.0,<2.12', '--pylint-extra-requirements=[]', '--pylint-lockfile=src/python/pants/backend/python/lint/pylint/pylint.lock', '--pylint-source-plugins=[]', "--backend-packages=+['pants.backend.python.lint.pylint']", '--yapf-version=yapf==0.32.0', "--yapf-extra-requirements=['toml']", '--yapf-lockfile=src/python/pants/backend/python/lint/yapf/yapf.lock', "--yapf-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.python.lint.yapf']", '--pyupgrade-version=pyupgrade>=2.31.0,<2.32', '--pyupgrade-extra-requirements=[]', '--pyupgrade-lockfile=src/python/pants/backend/python/lint/pyupgrade/pyupgrade.lock', "--pyupgrade-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.python.lint.pyupgrade']", '--ipython-version=ipython==7.16.1', '--ipython-extra-requirements=[]', '--ipython-lockfile=src/python/pants/backend/python/subsystems/ipython.lock', '--setuptools-version=setuptools>=50.3.0,<58.0', "--setuptools-extra-requirements=['wheel>=0.35.1,<0.38']", '--setuptools-lockfile=src/python/pants/backend/python/subsystems/setuptools.lock', '--setuptools-scm-version=setuptools-scm==6.4.2', '--setuptools-scm-extra-requirements=[]', '--setuptools-scm-lockfile=src/python/pants/backend/python/subsystems/setuptools_scm.lock', "--setuptools-scm-interpreter-constraints=['CPython>=3.7,<4']", '--sphinx-version=sphinx>=4.5.0,<4.6', "--sphinx-extra-requirements=['setuptools']", '--sphinx-lockfile=src/python/pants/backend/python/docs/sphinx/sphinx.lock', "--sphinx-interpreter-constraints=['CPython>=3.7,<4']", '--mypy-version=mypy==0.950', '--mypy-extra-requirements=[]', '--mypy-lockfile=src/python/pants/backend/python/typecheck/mypy/mypy.lock', "--mypy-interpreter-constraints=['CPython>=3.7,<4']", '--mypy-source-plugins=[]', '--mypy-protobuf-version=mypy-protobuf==2.10', '--mypy-protobuf-extra-requirements=[]', '--mypy-protobuf-lockfile=src/python/pants/backend/codegen/protobuf/python/mypy_protobuf.lock', "--mypy-protobuf-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.codegen.protobuf.python']", '--lambdex-version=lambdex==0.1.6', '--lambdex-extra-requirements=[]', '--lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock', "--lambdex-interpreter-constraints=['CPython>=3.7,<3.10']", "--backend-packages=+['pants.backend.awslambda.python']", '--pytest-version=pytest==7.0.1', "--pytest-extra-requirements=['pytest-cov>=2.12,!=2.12.1,<3.1']", '--pytest-lockfile=src/python/pants/backend/python/subsystems/pytest.lock', '--coverage-py-version=coverage[toml]>=5.5,<5.6', '--coverage-py-extra-requirements=[]', '--coverage-py-lockfile=src/python/pants/backend/python/subsystems/coverage_py.lock', "--coverage-py-interpreter-constraints=['CPython>=3.7,<4']", '--terraform-hcl2-parser-version=python-hcl2==3.0.5', '--terraform-hcl2-parser-extra-requirements=[]', '--terraform-hcl2-parser-lockfile=src/python/pants/backend/terraform/hcl2.lock', "--terraform-hcl2-parser-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.terraform']", '--dockerfile-parser-version=dockerfile==3.2.0', '--dockerfile-parser-extra-requirements=[]', '--dockerfile-parser-lockfile=src/python/pants/backend/docker/subsystems/dockerfile.lock', "--dockerfile-parser-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.docker']", '--twine-version=twine>=3.7.1,<3.8', "--twine-extra-requirements=['colorama>=0.4.3']", '--twine-lockfile=src/python/pants/backend/python/subsystems/twine.lock', "--twine-interpreter-constraints=['CPython>=3.7,<4']", '--clang-format-version=clang-format==14.0.3', '--clang-format-extra-requirements=[]', '--clang-format-lockfile=src/python/pants/backend/cc/lint/clangformat/clangformat.lock', "--clang-format-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.cc.lint.clangformat']", '--junit-version=5.7.2', "--junit-artifacts=('org.junit.platform:junit-platform-console:1.7.2', 'org.junit.jupiter:junit-jupiter-engine:{version}', 'org.junit.vintage:junit-vintage-engine:{version}')", '--junit-lockfile=src/python/pants/jvm/test/junit.default.lockfile.txt', '--google-java-format-version=1.13.0', "--google-java-format-artifacts=('com.google.googlejavaformat:google-java-format:{version}',)", '--google-java-format-lockfile=src/python/pants/backend/java/lint/google_java_format/google_java_format.default.lockfile.txt', '--scalafmt-version=3.2.1', "--scalafmt-artifacts=('org.scalameta:scalafmt-cli_2.13:{version}',)", '--scalafmt-lockfile=src/python/pants/backend/scala/lint/scalafmt/scalafmt.default.lockfile.txt', '--scalapb-version=0.11.6', "--scalapb-artifacts=('com.thesamet.scalapb:scalapbc_2.13:{version}',)", '--scalapb-lockfile=src/python/pants/backend/codegen/protobuf/scala/scalapbc.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.protobuf.scala']", '--scalatest-version=3.2.10', "--scalatest-artifacts=('org.scalatest:scalatest_2.13:{version}',)", '--scalatest-lockfile=src/python/pants/backend/scala/subsystems/scalatest.default.lockfile.txt', '--scrooge-version=21.12.0', "--scrooge-artifacts=('com.twitter:scrooge-generator_2.13:{version}',)", '--scrooge-lockfile=src/python/pants/backend/codegen/thrift/scrooge/scrooge.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.thrift.scrooge.scala']", '--java-avro-version=1.11.0', "--java-avro-artifacts=('org.apache.avro:avro-tools:{version}',)", '--java-avro-lockfile=src/python/pants/backend/codegen/avro/java/avro-tools.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.avro.java']", '--ktlint-version=0.45.2', "--ktlint-artifacts=('com.pinterest:ktlint:{version}',)", '--ktlint-lockfile=src/python/pants/backend/kotlin/lint/ktlint/ktlint.lock', "--backend-packages=+['pants.backend.experimental.kotlin.lint.ktlint']", 'generate-lockfiles', "--resolve=['lambdex']"]' returned non-zero exit status 1.

And for total clarity - odd numbers:

$ pkill pantsd
$ pgrep pantsd
$ PANTS_RG_LOOP_MAX=42 PANTS_RG_LOOP_THRESH=1137 ./build-support/bin/generate_all_lockfiles.sh --pex
thread '<unnamed>' panicked at 'Loop subgraph [thresh=1137, max=42]: digraph {
    0 [ label = "NodeIndex(18284): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(PathGlobs) -> Paths), in: (PathGlobs), out: ((DependenciesRequest, PathGlobs, UnexpandedTargets, Workspace))))" ]
...

So env vars do get through somehow. And even once the pantsd is already up:

$ ./pants -V
16:56:58.35 [INFO] Initializing scheduler...
16:56:58.69 [INFO] Scheduler initialized.
2.13.0.dev3
$ PANTS_RG_LOOP_MAX=33 PANTS_RG_LOOP_THRESH=1234 ./build-support/bin/generate_all_lockfiles.sh --pex
thread '<unnamed>' panicked at 'Loop subgraph [thresh=1234, max=33]: digraph {
...

[stuhood]

I get:

Your example lowers the value to 100000, which is what it was before the fix. And the ./build-support/bin/generate_all_lockfiles.sh script uses --no-concurrent, which disables pantsd, and will thus cause the env var to be observed.

[jsirois]

My what a tangle. Ok. Gotcha.

[jsirois]

@stuhood you're partially right. The complicating factor is there is at least some rule graph construction that occurs before daemonization and that can read env vars:

$ pkill pantsd
$ pgrep pantsd
$ PANTS_RG_LOOP_MAX=42 PANTS_RG_LOOP_THRESH=1137 ./pants --python-enable-resolves generate-lockfiles --resolve="['python-default']"
thread '<unnamed>' panicked at 'Loop subgraph [thresh=1137, max=42]: digraph {
...

That was not an issue in my original tests since even the default 5/100000 thresholds were enough to allow that initial rule graph construction to work. This example here uses such ridiculously low values, it doesn't.

A mazy tangle!