Closed jsirois closed 2 years ago
This bisects to 7eea60ed1378dab1de01787facfd363d35dc672b:
$ git bisect bad
7eea60ed1378dab1de01787facfd363d35dc672b is the first bad commit
commit 7eea60ed1378dab1de01787facfd363d35dc672b
Author: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
Date: Tue May 24 11:37:37 2022 -0500
Refactor `Specs` as prework for ignore Specs (#15620)
Prework for https://github.com/pantsbuild/pants/pull/15619. This does not actually wire up ignore specs yet, but it changes the `specs.py` modeling so that it's trivial to finish the feature.
As described at https://github.com/pantsbuild/pants/issues/15539, the semantics for ignore specs are simple: ignores always win out over includes.
Those semantics make modeling this problem easy. What used to be called `Specs` is now called `RawSpecs`: how to find all targets matching the specs, not considering ignores vs includes.
Then, we have a `Specs` object:
```python
@dataclass(frozen=True)
class Specs:
includes: RawSpecs = RawSpecs()
ignores: RawSpecs = RawSpecs()
And calculating the final `Addresses` is nothing more than:
```python
@rule
async def resolve_addresses_from_specs(specs: Specs) -> Addresses:
includes, ignores = await MultiGet(
Get(Addresses, RawSpecs, specs.includes),
Get(Addresses, RawSpecs, specs.ignores),
)
return Addresses(FrozenOrderedSet(includes) - FrozenOrderedSet(ignores))
```
When dealing directly with user specs, i.e. in goals, you should use `Specs` so ignores are included. Otherwise, rules can use either `Specs` or `RawSpecs` for things like `tailor` and dependency inference.
src/python/pants/backend/go/goals/tailor.py | 4 +- src/python/pants/backend/go/target_type_rules.py | 4 +- src/python/pants/backend/go/util_rules/go_mod.py | 4 +- src/python/pants/backend/project_info/peek_test.py | 6 +- .../pants/backend/python/goals/export_test.py | 6 +- src/python/pants/backend/python/goals/setup_py.py | 4 +- src/python/pants/backend/python/goals/tailor.py | 4 +- .../backend/terraform/dependency_inference.py | 4 +- src/python/pants/base/specs.py | 104 ++++++++++++------ src/python/pants/base/specs_parser.py | 63 ++++++++--- src/python/pants/base/specs_parser_test.py | 9 +- src/python/pants/base/specs_test.py | 20 ++-- src/python/pants/bin/local_pants_runner.py | 6 +- src/python/pants/bsp/util_rules/targets.py | 12 ++- src/python/pants/core/goals/tailor.py | 8 +- src/python/pants/core/goals/tailor_test.py | 16 +-- src/python/pants/engine/internals/graph.py | 16 +-- src/python/pants/engine/internals/specs_rules.py | 57 ++++++---- .../pants/engine/internals/specs_rules_test.py | 120 ++++++++++++--------- src/python/pants/engine/target.py | 2 +- src/python/pants/init/specs_calculator.py | 19 ++-- .../jvm/resolve/coursier_fetch_integration_test.py | 6 +- 22 files changed, 306 insertions(+), 188 deletions(-)
A (shlex-mangled) repro command looks like:
./pants --concurrent '--python-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --autoflake-version=autoflake==1.4 '--autoflake-extra-requirements=[]' --autoflake-lockfile=src/python/pants/backend/python/lint/autoflake/autoflake.lock '--autoflake-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--bandit-version=bandit>=1.7.0,<1.8' '--bandit-extra-requirements=['"'"'setuptools'"'"', '"'"'GitPython==3.1.18'"'"']' --bandit-lockfile=src/python/pants/backend/python/lint/bandit/bandit.lock '--backend-packages=+['"'"'pants.backend.python.lint.bandit'"'"']' --black-version=black==22.1.0 '--black-extra-requirements=['"'"'typing-extensions>=3.10.0.0; python_version < "3.10"'"'"']' --black-lockfile=src/python/pants/backend/python/lint/black/black.lock '--black-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--docformatter-version=docformatter>=1.4,<1.5' '--docformatter-extra-requirements=[]' --docformatter-lockfile=src/python/pants/backend/python/lint/docformatter/docformatter.lock '--docformatter-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--flake8-version=flake8>=3.9.2,<4.0' '--flake8-extra-requirements=[]' --flake8-lockfile=src/python/pants/backend/python/lint/flake8/flake8.lock '--flake8-source-plugins=[]' '--isort-version=isort[pyproject,colors]>=5.9.3,<6.0' '--isort-extra-requirements=[]' --isort-lockfile=src/python/pants/backend/python/lint/isort/isort.lock '--isort-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--pylint-version=pylint>=2.11.0,<2.12' '--pylint-extra-requirements=[]' --pylint-lockfile=src/python/pants/backend/python/lint/pylint/pylint.lock '--pylint-source-plugins=[]' '--backend-packages=+['"'"'pants.backend.python.lint.pylint'"'"']' --yapf-version=yapf==0.32.0 '--yapf-extra-requirements=['"'"'toml'"'"']' --yapf-lockfile=src/python/pants/backend/python/lint/yapf/yapf.lock '--yapf-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.python.lint.yapf'"'"']' '--pyupgrade-version=pyupgrade>=2.31.0,<2.32' '--pyupgrade-extra-requirements=[]' --pyupgrade-lockfile=src/python/pants/backend/python/lint/pyupgrade/pyupgrade.lock '--pyupgrade-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.experimental.python.lint.pyupgrade'"'"']' --ipython-version=ipython==7.16.1 '--ipython-extra-requirements=[]' --ipython-lockfile=src/python/pants/backend/python/subsystems/ipython.lock '--setuptools-version=setuptools>=50.3.0,<58.0' '--setuptools-extra-requirements=['"'"'wheel>=0.35.1,<0.38'"'"']' --setuptools-lockfile=src/python/pants/backend/python/subsystems/setuptools.lock --setuptools-scm-version=setuptools-scm==6.4.2 '--setuptools-scm-extra-requirements=[]' --setuptools-scm-lockfile=src/python/pants/backend/python/subsystems/setuptools_scm.lock '--setuptools-scm-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--sphinx-version=sphinx>=4.5.0,<4.6' '--sphinx-extra-requirements=['"'"'setuptools'"'"']' --sphinx-lockfile=src/python/pants/backend/python/docs/sphinx/sphinx.lock '--sphinx-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --mypy-version=mypy==0.950 '--mypy-extra-requirements=[]' --mypy-lockfile=src/python/pants/backend/python/typecheck/mypy/mypy.lock '--mypy-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--mypy-source-plugins=[]' --mypy-protobuf-version=mypy-protobuf==2.10 '--mypy-protobuf-extra-requirements=[]' --mypy-protobuf-lockfile=src/python/pants/backend/codegen/protobuf/python/mypy_protobuf.lock '--mypy-protobuf-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.codegen.protobuf.python'"'"']' --lambdex-version=lambdex==0.1.6 '--lambdex-extra-requirements=[]' --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock '--lambdex-interpreter-constraints=['"'"'CPython>=3.7,<3.10'"'"']' '--backend-packages=+['"'"'pants.backend.awslambda.python'"'"']' --pytest-version=pytest==7.0.1 '--pytest-extra-requirements=['"'"'pytest-cov>=2.12,!=2.12.1,<3.1'"'"']' --pytest-lockfile=src/python/pants/backend/python/subsystems/pytest.lock '--coverage-py-version=coverage[toml]>=5.5,<5.6' '--coverage-py-extra-requirements=[]' --coverage-py-lockfile=src/python/pants/backend/python/subsystems/coverage_py.lock '--coverage-py-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --terraform-hcl2-parser-version=python-hcl2==3.0.5 '--terraform-hcl2-parser-extra-requirements=[]' --terraform-hcl2-parser-lockfile=src/python/pants/backend/terraform/hcl2.lock '--terraform-hcl2-parser-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.experimental.terraform'"'"']' --dockerfile-parser-version=dockerfile==3.2.0 '--dockerfile-parser-extra-requirements=[]' --dockerfile-parser-lockfile=src/python/pants/backend/docker/subsystems/dockerfile.lock '--dockerfile-parser-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.docker'"'"']' '--twine-version=twine>=3.7.1,<3.8' '--twine-extra-requirements=['"'"'colorama>=0.4.3'"'"']' --twine-lockfile=src/python/pants/backend/python/subsystems/twine.lock '--twine-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' --clang-format-version=clang-format==14.0.3 '--clang-format-extra-requirements=[]' --clang-format-lockfile=src/python/pants/backend/cc/lint/clangformat/clangformat.lock '--clang-format-interpreter-constraints=['"'"'CPython>=3.7,<4'"'"']' '--backend-packages=+['"'"'pants.backend.experimental.cc.lint.clangformat'"'"']' --junit-version=5.7.2 '--junit-artifacts=('"'"'org.junit.platform:junit-platform-console:1.7.2'"'"', '"'"'org.junit.jupiter:junit-jupiter-engine:{version}'"'"', '"'"'org.junit.vintage:junit-vintage-engine:{version}'"'"')' --junit-lockfile=src/python/pants/jvm/test/junit.default.lockfile.txt --google-java-format-version=1.13.0 '--google-java-format-artifacts=('"'"'com.google.googlejavaformat:google-java-format:{version}'"'"',)' --google-java-format-lockfile=src/python/pants/backend/java/lint/google_java_format/google_java_format.default.lockfile.txt --scalafmt-version=3.2.1 '--scalafmt-artifacts=('"'"'org.scalameta:scalafmt-cli_2.13:{version}'"'"',)' --scalafmt-lockfile=src/python/pants/backend/scala/lint/scalafmt/scalafmt.default.lockfile.txt --scalapb-version=0.11.6 '--scalapb-artifacts=('"'"'com.thesamet.scalapb:scalapbc_2.13:{version}'"'"',)' --scalapb-lockfile=src/python/pants/backend/codegen/protobuf/scala/scalapbc.default.lockfile.txt '--backend-packages=+['"'"'pants.backend.experimental.codegen.protobuf.scala'"'"']' --scalatest-version=3.2.10 '--scalatest-artifacts=('"'"'org.scalatest:scalatest_2.13:{version}'"'"',)' --scalatest-lockfile=src/python/pants/backend/scala/subsystems/scalatest.default.lockfile.txt --scrooge-version=21.12.0 '--scrooge-artifacts=('"'"'com.twitter:scrooge-generator_2.13:{version}'"'"',)' --scrooge-lockfile=src/python/pants/backend/codegen/thrift/scrooge/scrooge.default.lockfile.txt '--backend-packages=+['"'"'pants.backend.experimental.codegen.thrift.scrooge.scala'"'"']' --java-avro-version=1.11.0 '--java-avro-artifacts=('"'"'org.apache.avro:avro-tools:{version}'"'"',)' --java-avro-lockfile=src/python/pants/backend/codegen/avro/java/avro-tools.default.lockfile.txt '--backend-packages=+['"'"'pants.backend.experimental.codegen.avro.java'"'"']' --ktlint-version=0.45.2 '--ktlint-artifacts=('"'"'com.pinterest:ktlint:{version}'"'"',)' --ktlint-lockfile=src/python/pants/backend/kotlin/lint/ktlint/ktlint.lock '--backend-packages=+['"'"'pants.backend.experimental.kotlin.lint.ktlint'"'"']' generate-lockfiles '--resolve=['"'"'lambdex'"'"']'
This is the minimal repro I find:
./pants --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock generate-lockfiles --resolve=lambdex
Remove any single backend (except 'pants.backend.awslambda.python' which is required for this), and things are green.
Ok, with this diff:
$ git diff src/rust/
diff --git a/src/rust/engine/rule_graph/src/builder.rs b/src/rust/engine/rule_graph/src/builder.rs
index e60873dfb..535e897ed 100644
--- a/src/rust/engine/rule_graph/src/builder.rs
+++ b/src/rust/engine/rule_graph/src/builder.rs
@@ -5,6 +5,7 @@ use crate::rules::{DependencyKey, ParamTypes, Query, Rule};
use crate::{params_str, Entry, EntryWithDeps, InnerEntry, RootEntry, RuleEdges, RuleGraph};
use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
+use std::env;
use indexmap::IndexSet;
use internment::Intern;
@@ -478,6 +479,7 @@ impl<R: Rule> Builder<R> {
let mut iteration = 0;
let mut maybe_in_loop = HashSet::new();
let mut looping = false;
+ let looping_threshold = env::var("PANTS_RG_LOOP_THRESH").map_or(100000, |v| v.parse::<usize>().unwrap());
while let Some(node_id) = to_visit.pop() {
let node = if let Some(node) = graph[node_id].inner() {
node
@@ -500,7 +502,7 @@ impl<R: Rule> Builder<R> {
}
iteration += 1;
- if iteration > 100000 {
+ if iteration > looping_threshold {
looping = true;
}
if iteration % 1000 == 0 {
@@ -665,7 +667,7 @@ impl<R: Rule> Builder<R> {
log::trace!("{}", trace_str);
maybe_in_loop.insert(node_id);
- if maybe_in_loop.len() > 5 {
+ if maybe_in_loop.len() > env::var("PANTS_RG_LOOP_MAX").map_or(5, |v| v.parse::<usize>().unwrap()) {
let subgraph = graph.filter_map(
|node_id, node| {
if maybe_in_loop.contains(&node_id) {
@@ -678,7 +680,8 @@ impl<R: Rule> Builder<R> {
);
panic!(
- "Loop subgraph: {}",
+ "Loop subgraph [{} in loop]: {}",
+ maybe_in_loop.len(),
petgraph::dot::Dot::with_config(&subgraph, &[])
);
}
I find non-deterministic success bumping up loop thresholds; so I'm not really sure if the bumps are effective or they just expose randomness better as they get bigger?:
Good:
$ PANTS_RG_LOOP_MAX=6 PANTS_RG_LOOP_THRESH=1000000000000000 ./pants --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock generate-lockfiles --resolve=lambdex
19:40:13.87 [INFO] Initializing scheduler...
19:40:14.23 [INFO] Scheduler initialized.
19:40:18.31 [INFO] Completed: Generate lockfile for lambdex
19:40:18.31 [INFO] Wrote lockfile for the resolve `lambdex` to src/python/pants/backend/python/subsystems/lambdex.lock
Then pkill pantsd
and make sure all pantsd are down. Then:
$ PANTS_RG_LOOP_MAX=6 PANTS_RG_LOOP_THRESH=1000000000000000 ./pants --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" --lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock generate-lockfiles --resolve=lambdex
19:40:25.91 [INFO] Initializing scheduler...
19:40:26.25 [ERROR] panic at 'Loop subgraph [6 in loop]: digraph {
0 [ label = "NodeIndex(49419): ParamsLabeled(node: @rule(pants.backend.scala.dependency_inference.rules:165:inject_scala_plugin_dependencies(InjectScalaPluginDependenciesRequest) -> InjectedDependencies, gets=[Get(WrappedTarget, Address), Get(ScalaPluginTargetsForTarget, ScalaPluginsForTargetWithoutResolveRequest)]), in: (InjectScalaPluginDependenciesRequest), out: (InjectScalaPluginDependenciesRequest))" ]
1 [ label = "NodeIndex(79211): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.options_parsing:38:parse_options(BuildConfiguration, SessionValues) -> _Options), in: (()), out: (())))" ]
2 [ label = "NodeIndex(79212): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.options_parsing:46:scope_options(Scope, _Options) -> ScopedOptions), in: (Scope), out: (())))" ]
3 [ label = "NodeIndex(79213): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.option.global_options:1446:construct_scope_() -> GlobalOptions, gets=[Get(ScopedOptions, Scope)]), in: (()), out: (())))" ]
4 [ label = "NodeIndex(79220): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.graph:108:target_types_to_generate_targets_requests(UnionMembership) -> TargetTypesToGenerateTargetsRequests), in: (()), out: (())))" ]
5 [ label = "NodeIndex(79222): Deleted(reason: NoDependees, ParamsLabeled(node: @rule(pants.engine.internals.graph:734:extract_unmatched_build_file_globs(GlobalOptions) -> UnmatchedBuildFileGlobs), in: (()), out: (())))" ]
2 -> 1 [ label = "_Options" ]
3 -> 2 [ label = "Get(ScopedOptions, Scope)" ]
5 -> 3 [ label = "GlobalOptions" ]
}
', /home/jsirois/dev/pantsbuild/jsirois-pants/src/rust/engine/rule_graph/src/builder.rs:682
19:40:26.25 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
^CTraceback (most recent call last):
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 119, in <module>
main()
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 115, in main
PantsLoader.main()
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 111, in main
cls.run_default_entrypoint()
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
exit_code = runner.run(start_time)
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_runner.py", line 89, in run
return remote_runner.run(start_time)
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/remote_pants_runner.py", line 117, in run
return self._connect_and_execute(pantsd_handle, start_time)
File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/remote_pants_runner.py", line 151, in _connect_and_execute
return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.
If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.
If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (https://www.pantsbuild.org/docs/getting-help).
I find non-deterministic success bumping up loop thresholds; so I'm not really sure if the bumps are effective or they just expose randomness better as they get bigger?:
I see deterministic success with a higher "looping" threshold... I think that you might have been running into the fact that we prune all env vars in pantsd
?
With your patch applied and --no-pantsd
, this command reliably succeeds:
PANTS_RG_LOOP_THRESH=1000000 ./pants --no-pantsd --backend-packages="+['pants.backend.awslambda.python','pants.backend.python.lint.bandit','pants.backend.python.lint.pylint','pants.backend.python.lint.yapf','pants.backend.experimental.python.lint.pyupgrade','pants.backend.codegen.protobuf.python']" help
Given the env var filtering issue, and since I'm hoping to be able to resume #11269 this summer, I'll likely bump the hardcoded loop threshold for now.
Can't be. Are they pruned for the initial pantsd bring up? As noted I killed pantsd between all attempts.
Are they pruned for the initial pantsd bring up?
Yes, because pantsd
is always forked, and truncates its own env vars as it starts up: https://github.com/pantsbuild/pants/blob/b251df7105ff597031196ab0c85800905727c59a/src/python/pants/pantsd/pants_daemon.py#L176-L181
So my success vs failure was non-deterministic using existing thresholds. Is that expected? That the algorithm is non-deterministic?
I'm not sure. But I suspect that one of your successful attempts was using an outdated instance of pantsd
where the value was hardcoded, perhaps (assuming that you went through a few iterations before arriving at that patch)...?
Is that expected? That the algorithm is non-deterministic?
It is expected in that we are using Rust's default randomized hash function... but I've never known it to affect success or failure. #11269 switches to deterministic hashing for that and other reasons.
On main after your fix ./build-support/bin/generate_all_lockfiles.sh --pex
is green.
But, with ~my diff re-applied:
$ git diff src/rust/
diff --git a/src/rust/engine/rule_graph/src/builder.rs b/src/rust/engine/rule_graph/src/builder.rs
index 4c9ef3141..2de045e36 100644
--- a/src/rust/engine/rule_graph/src/builder.rs
+++ b/src/rust/engine/rule_graph/src/builder.rs
@@ -5,6 +5,7 @@ use crate::rules::{DependencyKey, ParamTypes, Query, Rule};
use crate::{params_str, Entry, EntryWithDeps, InnerEntry, RootEntry, RuleEdges, RuleGraph};
use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
+use std::env;
use indexmap::IndexSet;
use internment::Intern;
@@ -478,6 +479,9 @@ impl<R: Rule> Builder<R> {
let mut iteration = 0;
let mut maybe_in_loop = HashSet::new();
let mut looping = false;
+ let loop_thresh =
+ env::var("PANTS_RG_LOOP_THRESH").map_or(10000000, |v| v.parse::<usize>().unwrap());
+ let loop_max = env::var("PANTS_RG_LOOP_MAX").map_or(5, |v| v.parse::<usize>().unwrap());
while let Some(node_id) = to_visit.pop() {
let node = if let Some(node) = graph[node_id].inner() {
node
@@ -507,7 +511,7 @@ impl<R: Rule> Builder<R> {
// See https://github.com/pantsbuild/pants/issues/11269 for plans to improve this
// implementation.
iteration += 1;
- if iteration > 10000000 {
+ if iteration > loop_thresh {
looping = true;
}
if iteration % 1000 == 0 {
@@ -672,7 +676,7 @@ impl<R: Rule> Builder<R> {
log::trace!("{}", trace_str);
maybe_in_loop.insert(node_id);
- if maybe_in_loop.len() > 5 {
+ if maybe_in_loop.len() > loop_max {
let subgraph = graph.filter_map(
|node_id, node| {
if maybe_in_loop.contains(&node_id) {
@@ -685,7 +689,9 @@ impl<R: Rule> Builder<R> {
);
panic!(
- "Loop subgraph: {}",
+ "Loop subgraph [thresh={}, max={}]: {}",
+ loop_thresh,
+ loop_max,
petgraph::dot::Dot::with_config(&subgraph, &[])
);
}
I get:
$ pkill pantsd
$ pgrep pantsd
$ PANTS_RG_LOOP_MAX=5 PANTS_RG_LOOP_THRESH=100000 ./build-support/bin/generate_all_lockfiles.sh --pex
16:44:37.90 [INFO] Initializing scheduler...
16:44:38.25 [INFO] Scheduler initialized.
16:45:05.80 [INFO] Completed: Generate lockfile for python-default
16:45:05.81 [INFO] Wrote lockfile for the resolve `python-default` to 3rdparty/python/user_reqs.lock
16:45:08.07 [ERROR] panic at 'Loop subgraph [thresh=100000, max=5]: digraph {
0 [ label = "NodeIndex(23432): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.core.util_rules.archive:98:convert_digest_to_MaybeExtractArchiveRequest(Digest) -> MaybeExtractArchiveRequest), in: (Digest), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest))))" ]
1 [ label = "NodeIndex(23433): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(Digest) -> Snapshot), in: (Digest), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest))))" ]
2 [ label = "NodeIndex(23439): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(RemovePrefix) -> Digest), in: (RemovePrefix), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest, RemovePrefix))))" ]
3 [ label = "NodeIndex(23446): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(MergeDigests) -> Digest), in: (MergeDigests), out: ((Digest, ExportPythonTool, MergeDigests, OptionalPexRequest, PexRequest))))" ]
4 [ label = "NodeIndex(23451): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(CreateDigest) -> Digest), in: (CreateDigest), out: ((CreateDigest, Digest, ExportPythonTool, OptionalPexRequest, PexRequest))))" ]
5 [ label = "NodeIndex(23568): ParamsLabeled(node: @rule(<intrinsic>(Process) -> FallibleProcessResult), in: (Process), out: ((Digest, ExportPythonTool, OptionalPexRequest, PexRequest, Process, UnzipBinaryRequest)))" ]
}
', /home/jsirois/dev/pantsbuild/pants/src/rust/engine/rule_graph/src/builder.rs:691
16:45:08.07 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
Exception caught: (pyo3_runtime.PanicException)
...
subprocess.CalledProcessError: Command '['./pants', '--concurrent', "--python-interpreter-constraints=['CPython>=3.7,<4']", '--autoflake-version=autoflake==1.4', '--autoflake-extra-requirements=[]', '--autoflake-lockfile=src/python/pants/backend/python/lint/autoflake/autoflake.lock', "--autoflake-interpreter-constraints=['CPython>=3.7,<4']", '--bandit-version=bandit>=1.7.0,<1.8', "--bandit-extra-requirements=['setuptools', 'GitPython==3.1.18']", '--bandit-lockfile=src/python/pants/backend/python/lint/bandit/bandit.lock', "--backend-packages=+['pants.backend.python.lint.bandit']", '--black-version=black==22.1.0', '--black-extra-requirements=[\'typing-extensions>=3.10.0.0; python_version < "3.10"\']', '--black-lockfile=src/python/pants/backend/python/lint/black/black.lock', "--black-interpreter-constraints=['CPython>=3.7,<4']", '--docformatter-version=docformatter>=1.4,<1.5', '--docformatter-extra-requirements=[]', '--docformatter-lockfile=src/python/pants/backend/python/lint/docformatter/docformatter.lock', "--docformatter-interpreter-constraints=['CPython>=3.7,<4']", '--flake8-version=flake8>=3.9.2,<4.0', '--flake8-extra-requirements=[]', '--flake8-lockfile=src/python/pants/backend/python/lint/flake8/flake8.lock', '--flake8-source-plugins=[]', '--isort-version=isort[pyproject,colors]>=5.9.3,<6.0', '--isort-extra-requirements=[]', '--isort-lockfile=src/python/pants/backend/python/lint/isort/isort.lock', "--isort-interpreter-constraints=['CPython>=3.7,<4']", '--pylint-version=pylint>=2.11.0,<2.12', '--pylint-extra-requirements=[]', '--pylint-lockfile=src/python/pants/backend/python/lint/pylint/pylint.lock', '--pylint-source-plugins=[]', "--backend-packages=+['pants.backend.python.lint.pylint']", '--yapf-version=yapf==0.32.0', "--yapf-extra-requirements=['toml']", '--yapf-lockfile=src/python/pants/backend/python/lint/yapf/yapf.lock', "--yapf-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.python.lint.yapf']", '--pyupgrade-version=pyupgrade>=2.31.0,<2.32', '--pyupgrade-extra-requirements=[]', '--pyupgrade-lockfile=src/python/pants/backend/python/lint/pyupgrade/pyupgrade.lock', "--pyupgrade-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.python.lint.pyupgrade']", '--ipython-version=ipython==7.16.1', '--ipython-extra-requirements=[]', '--ipython-lockfile=src/python/pants/backend/python/subsystems/ipython.lock', '--setuptools-version=setuptools>=50.3.0,<58.0', "--setuptools-extra-requirements=['wheel>=0.35.1,<0.38']", '--setuptools-lockfile=src/python/pants/backend/python/subsystems/setuptools.lock', '--setuptools-scm-version=setuptools-scm==6.4.2', '--setuptools-scm-extra-requirements=[]', '--setuptools-scm-lockfile=src/python/pants/backend/python/subsystems/setuptools_scm.lock', "--setuptools-scm-interpreter-constraints=['CPython>=3.7,<4']", '--sphinx-version=sphinx>=4.5.0,<4.6', "--sphinx-extra-requirements=['setuptools']", '--sphinx-lockfile=src/python/pants/backend/python/docs/sphinx/sphinx.lock', "--sphinx-interpreter-constraints=['CPython>=3.7,<4']", '--mypy-version=mypy==0.950', '--mypy-extra-requirements=[]', '--mypy-lockfile=src/python/pants/backend/python/typecheck/mypy/mypy.lock', "--mypy-interpreter-constraints=['CPython>=3.7,<4']", '--mypy-source-plugins=[]', '--mypy-protobuf-version=mypy-protobuf==2.10', '--mypy-protobuf-extra-requirements=[]', '--mypy-protobuf-lockfile=src/python/pants/backend/codegen/protobuf/python/mypy_protobuf.lock', "--mypy-protobuf-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.codegen.protobuf.python']", '--lambdex-version=lambdex==0.1.6', '--lambdex-extra-requirements=[]', '--lambdex-lockfile=src/python/pants/backend/python/subsystems/lambdex.lock', "--lambdex-interpreter-constraints=['CPython>=3.7,<3.10']", "--backend-packages=+['pants.backend.awslambda.python']", '--pytest-version=pytest==7.0.1', "--pytest-extra-requirements=['pytest-cov>=2.12,!=2.12.1,<3.1']", '--pytest-lockfile=src/python/pants/backend/python/subsystems/pytest.lock', '--coverage-py-version=coverage[toml]>=5.5,<5.6', '--coverage-py-extra-requirements=[]', '--coverage-py-lockfile=src/python/pants/backend/python/subsystems/coverage_py.lock', "--coverage-py-interpreter-constraints=['CPython>=3.7,<4']", '--terraform-hcl2-parser-version=python-hcl2==3.0.5', '--terraform-hcl2-parser-extra-requirements=[]', '--terraform-hcl2-parser-lockfile=src/python/pants/backend/terraform/hcl2.lock', "--terraform-hcl2-parser-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.terraform']", '--dockerfile-parser-version=dockerfile==3.2.0', '--dockerfile-parser-extra-requirements=[]', '--dockerfile-parser-lockfile=src/python/pants/backend/docker/subsystems/dockerfile.lock', "--dockerfile-parser-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.docker']", '--twine-version=twine>=3.7.1,<3.8', "--twine-extra-requirements=['colorama>=0.4.3']", '--twine-lockfile=src/python/pants/backend/python/subsystems/twine.lock', "--twine-interpreter-constraints=['CPython>=3.7,<4']", '--clang-format-version=clang-format==14.0.3', '--clang-format-extra-requirements=[]', '--clang-format-lockfile=src/python/pants/backend/cc/lint/clangformat/clangformat.lock', "--clang-format-interpreter-constraints=['CPython>=3.7,<4']", "--backend-packages=+['pants.backend.experimental.cc.lint.clangformat']", '--junit-version=5.7.2', "--junit-artifacts=('org.junit.platform:junit-platform-console:1.7.2', 'org.junit.jupiter:junit-jupiter-engine:{version}', 'org.junit.vintage:junit-vintage-engine:{version}')", '--junit-lockfile=src/python/pants/jvm/test/junit.default.lockfile.txt', '--google-java-format-version=1.13.0', "--google-java-format-artifacts=('com.google.googlejavaformat:google-java-format:{version}',)", '--google-java-format-lockfile=src/python/pants/backend/java/lint/google_java_format/google_java_format.default.lockfile.txt', '--scalafmt-version=3.2.1', "--scalafmt-artifacts=('org.scalameta:scalafmt-cli_2.13:{version}',)", '--scalafmt-lockfile=src/python/pants/backend/scala/lint/scalafmt/scalafmt.default.lockfile.txt', '--scalapb-version=0.11.6', "--scalapb-artifacts=('com.thesamet.scalapb:scalapbc_2.13:{version}',)", '--scalapb-lockfile=src/python/pants/backend/codegen/protobuf/scala/scalapbc.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.protobuf.scala']", '--scalatest-version=3.2.10', "--scalatest-artifacts=('org.scalatest:scalatest_2.13:{version}',)", '--scalatest-lockfile=src/python/pants/backend/scala/subsystems/scalatest.default.lockfile.txt', '--scrooge-version=21.12.0', "--scrooge-artifacts=('com.twitter:scrooge-generator_2.13:{version}',)", '--scrooge-lockfile=src/python/pants/backend/codegen/thrift/scrooge/scrooge.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.thrift.scrooge.scala']", '--java-avro-version=1.11.0', "--java-avro-artifacts=('org.apache.avro:avro-tools:{version}',)", '--java-avro-lockfile=src/python/pants/backend/codegen/avro/java/avro-tools.default.lockfile.txt', "--backend-packages=+['pants.backend.experimental.codegen.avro.java']", '--ktlint-version=0.45.2', "--ktlint-artifacts=('com.pinterest:ktlint:{version}',)", '--ktlint-lockfile=src/python/pants/backend/kotlin/lint/ktlint/ktlint.lock', "--backend-packages=+['pants.backend.experimental.kotlin.lint.ktlint']", 'generate-lockfiles', "--resolve=['lambdex']"]' returned non-zero exit status 1.
And for total clarity - odd numbers:
$ pkill pantsd
$ pgrep pantsd
$ PANTS_RG_LOOP_MAX=42 PANTS_RG_LOOP_THRESH=1137 ./build-support/bin/generate_all_lockfiles.sh --pex
thread '<unnamed>' panicked at 'Loop subgraph [thresh=1137, max=42]: digraph {
0 [ label = "NodeIndex(18284): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(<intrinsic>(PathGlobs) -> Paths), in: (PathGlobs), out: ((DependenciesRequest, PathGlobs, UnexpandedTargets, Workspace))))" ]
...
So env vars do get through somehow. And even once the pantsd is already up:
$ ./pants -V
16:56:58.35 [INFO] Initializing scheduler...
16:56:58.69 [INFO] Scheduler initialized.
2.13.0.dev3
$ PANTS_RG_LOOP_MAX=33 PANTS_RG_LOOP_THRESH=1234 ./build-support/bin/generate_all_lockfiles.sh --pex
thread '<unnamed>' panicked at 'Loop subgraph [thresh=1234, max=33]: digraph {
...
I get:
Your example lowers the value to 100000
, which is what it was before the fix. And the ./build-support/bin/generate_all_lockfiles.sh
script uses --no-concurrent
, which disables pantsd
, and will thus cause the env var to be observed.
My what a tangle. Ok. Gotcha.
@stuhood you're partially right. The complicating factor is there is at least some rule graph construction that occurs before daemonization and that can read env vars:
$ pkill pantsd
$ pgrep pantsd
$ PANTS_RG_LOOP_MAX=42 PANTS_RG_LOOP_THRESH=1137 ./pants --python-enable-resolves generate-lockfiles --resolve="['python-default']"
thread '<unnamed>' panicked at 'Loop subgraph [thresh=1137, max=42]: digraph {
...
That was not an issue in my original tests since even the default 5/100000 thresholds were enough to allow that initial rule graph construction to work. This example here uses such ridiculously low values, it doesn't.
A mazy tangle!
On main @ d39465be54c59af3b2650bdc245431d440795fc0 I consistently see a Loop subgraph panic from here: https://github.com/pantsbuild/pants/blob/d39465be54c59af3b2650bdc245431d440795fc0/src/rust/engine/rule_graph/src/builder.rs#L680-L683