search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.35k stars 640 forks source link

"Too many open files" errors on 2.17 #19765

Closed danxmoran closed 1 year ago

danxmoran commented 1 year ago

Describe the bug

While testing out v2.17.0, I ran the following to test the perf of Rust-based dependency inference:

pants --no-local-cache --no-pantsd peek ::

This failed with:

11:56:35.57 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.backend.project_info.peek.peek
    `peek` goal

Traceback (most recent call last):
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/backend/project_info/peek.py", line 256, in peek
    tds = await Get(TargetDatas, UnexpandedTargets, targets)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/backend/project_info/peek.py", line 186, in get_target_data
    hydrated_sources_per_target = await MultiGet(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 361, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 168, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 1139, in hydrate_sources
    snapshot = await Get(Snapshot, PathGlobs, path_globs)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
native_engine.IntrinsicError: Snapshot failed: Failed to digest inputs: Throw { val: Failed to open "/Users/dan.moran/color/src/projects/common/frontend/components/src/Notifications/useToast/useToast.tsx": Too many open files (os error 24), python_traceback: "Traceback (no traceback):\n  <pants native internals>\nException: Failed to open \"/Users/dan.moran/color/src/projects/common/frontend/components/src/Notifications/useToast/useToast.tsx\": Too many open files (os error 24)", engine_traceback: [FailureFrame { name: "digest_file", desc: Some("Fingerprinting: src/projects/common/frontend/components/src/Notifications/useToast/useToast.tsx") }] }

I thought it might be a problem with the new Rust-based system, so I ran the following to find the environment variable that would revert back to the Python-based implementation:

pants help-all

But it also crashed with:

14:27:34.19 [ERROR] [Errno 24] Too many open files
Traceback (most recent call last):
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/bin/daemon_pants_runner.py", line 142, in single_daemonized_run
    return runner.run(start_time)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 271, in run
    engine_result = self._run_inner()
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 230, in _run_inner
    return self._run_builtin_goal(self.options.builtin_goal)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 220, in _run_builtin_goal
    return goal.run(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/goal/help.py", line 61, in run
    return help_printer.print_help()
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_printer.py", line 67, in print_help
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_printer.py", line 171, in _print_all_help
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_printer.py", line 583, in _get_help_json
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_info_extracter.py", line 428, in asdict
    return {
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_info_extracter.py", line 429, in <dictcomp>
    field: {thing: dataclasses.asdict(info) for thing, info in value.items()}
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_info_extracter.py", line 429, in <dictcomp>
    field: {thing: dataclasses.asdict(info) for thing, info in value.items()}
  File "/Users/dan.moran/Library/Caches/nce/3abc4d5fbbc80f5f848f280927ac5d13de8dc03aabb6ae65d8247cbb68e6f6bf/cpython-3.9.16+20230507-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/_collections_abc.py", line 851, in __iter__
    yield (key, self._mapping[key])
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/util/frozendict.py", line 149, in __getitem__
    return self._get_value(k)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/util/memo.py", line 123, in memoize
    result = func(*args, **kwargs)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/util/frozendict.py", line 153, in _get_value
    return cast("Callable[[], V]", self._data[k])()
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/help/help_info_extracter.py", line 454, in load
    options.for_scope(scope_info.scope)  # Force parsing.
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/util/memo.py", line 123, in memoize
    result = func(*args, **kwargs)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/options.py", line 350, in for_scope
    values = self.get_parser(scope).parse_args(parse_args_request)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/parser.py", line 247, in parse_args
    value_history = self._compute_value(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/parser.py", line 607, in _compute_value
    config_source_files = self._config.get_sources_for_option(config_section, dest)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/config.py", line 168, in get_sources_for_option
    paths.append(os.path.relpath(vals.path))
  File "/Users/dan.moran/Library/Caches/nce/3abc4d5fbbc80f5f848f280927ac5d13de8dc03aabb6ae65d8247cbb68e6f6bf/cpython-3.9.16+20230507-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/posixpath.py", line 472, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/Users/dan.moran/Library/Caches/nce/3abc4d5fbbc80f5f848f280927ac5d13de8dc03aabb6ae65d8247cbb68e6f6bf/cpython-3.9.16+20230507-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/posixpath.py", line 380, in abspath
    cwd = os.getcwd()
OSError: [Errno 24] Too many open files

Strangely, toggling pantsd on/off fixes the problem, but in different directions for the two goals:

# Both of these work
pants --no-local-cache peek ::
pants --no-pantsd help-all

Pants version

2.17.0

OS

MacOS

danxmoran commented 1 year ago

Hmmm also seeing with fix. This crashes:

pants fix ::

With:

Traceback (most recent call last):
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/core/goals/fix.py", line 348, in fix
    return await _do_fix(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/core/goals/fix.py", line 273, in _do_fix
    partitions_by_request_type = await _get_partitions_by_request_type(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/core/goals/lint.py", line 368, in _get_partitions_by_request_type
    targets, specs_paths = await MultiGet(_get_targets, _get_specs_paths)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 512, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 168, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 272, in resolve_addresses_from_specs
    includes, ignores = await MultiGet(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 512, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 168, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 260, in resolve_addresses_from_raw_specs
    without_file_owners, with_file_owners = await MultiGet(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 512, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 168, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 179, in addresses_from_raw_specs_without_file_owners
    address_families = await Get(AddressFamilies, RawSpecsWithoutFileOwners, specs)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 149, in address_families_from_raw_specs_without_file_owners
    await Get(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/synthetic_targets.py", line 298, in all_synthetic_targets
    all_synthetic = await MultiGet(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 361, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 168, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/subsystem.py", line 311, in _construct_subsytem
    scoped_options = await Get(ScopedOptions, Scope(str(subsystem_typ.options_scope)))
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/parser.py", line 247, in parse_args
    value_history = self._compute_value(
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/parser.py", line 607, in _compute_value
    config_source_files = self._config.get_sources_for_option(config_section, dest)
  File "/Users/dan.moran/Library/Caches/nce/50b60c9f29c54578592fd2b4725722435f65d43cb21e6e2732268b6d5050a09a/bindings/venvs/2.17.0/lib/python3.9/site-packages/pants/option/config.py", line 168, in get_sources_for_option
    paths.append(os.path.relpath(vals.path))
  File "/Users/dan.moran/Library/Caches/nce/3abc4d5fbbc80f5f848f280927ac5d13de8dc03aabb6ae65d8247cbb68e6f6bf/cpython-3.9.16+20230507-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/posixpath.py", line 472, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/Users/dan.moran/Library/Caches/nce/3abc4d5fbbc80f5f848f280927ac5d13de8dc03aabb6ae65d8247cbb68e6f6bf/cpython-3.9.16+20230507-x86_64-apple-darwin-install_only.tar.gz/python/lib/python3.9/posixpath.py", line 380, in abspath
    cwd = os.getcwd()
OSError: [Errno 24] Too many open files

But this runs fine:

pants --no-pantsd fix ::
huonw commented 1 year ago

I thought it might be a problem with the new Rust-based system

Good thinking: the env var is PANTS_PYTHON_INFER_USE_RUST_PARSER=false (or editing pants.toml to set [python-infer] use_rust_parser = false temporarily). Does it reproduce with that set?

(You may need to pkill -f pantsd to ensure that pantsd is starting fresh and doesn't have any left over file-handles hanging around.)

danxmoran commented 1 year ago

@huonw I'm no longer able to consistently repro the problem on 2.17 (it still happens often, but not every time). I can say that I haven't seen the error yet when using PANTS_PYTHON_INFER_USE_RUST_PARSER=false

stuhood commented 1 year ago

The rust based parser consumes snapshots/digests (as did the Python parser, albeit via forked processes), and so shouldn't involve any "new" sources of open file handles.

@danxmoran : What is ulimit -n on the relevant machine?

danxmoran commented 1 year ago 
$ ulimit -n
256

Could the problem be from a difference in concurrency limits? The Python parser would have been constrained in some way by [GLOBAL].process_execution_local_parallelism. Is there a similar limit for the Rust parser?

benjyw commented 1 year ago

You've read this presumably? Your ulimit is very low indeed.

danxmoran commented 1 year ago

Your ulimit is very low indeed.

That may be true! But even with Color's large codebase I haven't needed to bump the limit before 2.17, and it's especially fishy (to me, not knowing all the details of how it works) that goals like help-all would run into the error. So while I can configure my shell to automatically raise the limit, it feels like something substantial has regressed/changed in the latest release.

stuhood commented 1 year ago

Could the problem be from a difference in concurrency limits? The Python parser would have been constrained in some way by [GLOBAL].process_execution_local_parallelism. Is there a similar limit for the Rust parser?

No, because in both cases, the way that those parsers actually get files to operate on is by capturing Snapshots of them. If anything, because the Rust parser doesn't put the files on disk before parsing them, it should be using fewer file handles.

One thing that you could do is compare lsof -p $pid for the two different versions (although "when" you compare matters), and see which files are open.

sureshjoshi commented 1 year ago

So it's not lost to the ether, from Slack

adhoc_tool(
    name="node-modules",
    runnable=":pnpm", 
    args=["install", "--frozen-lockfile"], 
    runnable_dependencies=[":node", ":sh"],
    execution_dependencies=[":build-meta"],
    output_directories=["node_modules"],
    timeout=300,
)

Leads to this too many open files error.

16:24:09.34 [INFO] Completed: Running the `adhoc_tool` at frontend/web:node-modules
16:24:09.34 [ERROR] 1 Exception encountered:

Engine traceback:
  in `run` goal

IntrinsicError: Failed to execute: Process {
    argv: [
        "/bin/bash",
        "-c",
        "cd frontend/web && /opt/homebrew/bin/pnpm install --frozen-lockfile",
    ],
    env: {
        "PATH": "{chroot}/_runnable_dependency_shims_1ec700094d6da0d2a2f0d61fd955b4abd702f3190cb7829fb9beee4656c42c14",
        "_PANTS_SHIM_ROOT": "{chroot}",
    },
    working_directory: None,
    input_digests: InputDigests {
        complete: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<d4d387daf2f27e12bac6d951ef5a014f75320b05b68f9b9ecdf4a4805b065e62>,
                size_bytes: 249,
            },
            tree: "Some(..)",
        },
        nailgun: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        inputs: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<9873767679ce16f69603180b0efeeeb69e02ec58b8a5d6c97ac26c5a67b52d3b>,
                size_bytes: 82,
            },
            tree: "Some(..)",
        },
        immutable_inputs: {
            RelativePath(
                "_runnable_dependency_shims_1ec700094d6da0d2a2f0d61fd955b4abd702f3190cb7829fb9beee4656c42c14",
            ): DirectoryDigest {
                digest: Digest {
                    hash: Fingerprint<1ec700094d6da0d2a2f0d61fd955b4abd702f3190cb7829fb9beee4656c42c14>,
                    size_bytes: 158,
                },
                tree: "Some(..)",
            },
        },
        use_nailgun: {},
    },
    output_files: {},
    output_directories: {
        RelativePath(
            "frontend/web/node_modules",
        ),
    },
    timeout: None,
    execution_slot_variable: None,
    concurrency_available: 0,
    description: "Running the `adhoc_tool` at frontend/web:node-modules",
    level: Info,
    append_only_caches: {},
    jdk_home: None,
    cache_scope: Successful,
    execution_environment: ProcessExecutionEnvironment {
        name: None,
        platform: Macos_arm64,
        strategy: Local,
    },
    remote_cache_speculation_delay: 0ns,
}

Failed to digest inputs: "Failed to open \"/private/var/folders/dy/q08y_dts5vd71rm99t4gc9lr0000gp/T/pants-sandbox-p1U8nu/frontend/web/node_modules/.pnpm/@typescript-eslint+eslint-plugin@6.7.3_@typescript-eslint+parser@6.7.3_eslint@8.50.0_typescript@5.2.2/node_modules/@typescript-eslint/eslint-plugin/docs/rules/no-confusing-void-expression.md\": Too many open files (os error 24)"

Can be re-produced on pantsbuild/example-adhoc javascript example by replacing the deps with:

 "dependencies": {
    "algoliasearch": "^4.17.1",
    "autoprefixer": "^10.4.16",
    "cssnano": "^6.0.1",
    "date-fns": "^2.30.0",  
    "eslint": "^8.50.0",
    "eslint-config-prettier": "^9.0.0",
    "eslint-plugin-jest-dom": "^5.1.0",
    "eslint-plugin-square-svelte-store": "^1.0.0",
    "eslint-plugin-svelte": "^2.33.2",
    "eslint-plugin-testing-library": "^6.0.2",
    "firebase": "^9.22.1",
    "hammerjs": "^2.0.8",
    "happy-dom": "^12.2.1",
    "immer": "^10.0.2",
    "msw": "^1.3.1",
    "postcss": "^8.4.31",
    "es-leftpad": "^1.0.0",
    "prettier": "^3.0.3",
    "prettier-plugin-organize-imports": "^3.2.3",
    "prettier-plugin-svelte": "^3.0.3",
    "prettier-plugin-tailwindcss": "^0.5.4",
    "svelte": "^4.2.1",
    "svelte-check": "^3.5.2",
    "svelte-i18n": "^3.7.4",
    "tailwindcss": "^3.3.3",
    "ts-node": "^10.9.1",
    "tslib": "^2.6.2",
    "typescript": "^5.2.2",
    "vite": "^4.4.9",
    "vitest": "^0.34.6"

and running pants run :run-js-app

benjyw commented 1 year ago

I did some experimenting, and @sureshjoshi's specific case is due to trying to hold open over 9500 files under node_modules. Lots of big stuff in there (firebase alone is ~5500 files).

benjyw commented 1 year ago

To clarify, this is pantsd holding the files open, presumably while materializing the sandbox, not node (which hasn't run yet).

sureshjoshi commented 1 year ago

From Stu's PR, it looks like it's related to concurrent access to what are now files (previously in the LMDB store).

I thought that was only for "large" files - but in any case, I'd call this a pretty small reproduction repo. They get far nuttier very quickly.

benjyw commented 1 year ago

Can confirm that the javascript repro above is fixed by #20055