search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.29k stars 631 forks source link

Adding `--level=error` to xargs-piped pants command causes internal exception #18845

Open engnatha opened 1 year ago

engnatha commented 1 year ago

Describe the bug I can run the following command and the pants logs look great and everything happens fast.

./pants dependees list --changed-since=master | xargs ./pants dependees list | xargs ./pants --filter-target-type=python_test list

If I add --level=error to the last command to change the log level, I get exceptions in the pants logs.

Unfortunately, no amount of RUST_BACKTRACE=1 setting in my environment was able to show the full details (maybe an issue with xargs?).

Both variants of the command produce the same result, but the latter causes some performance issues since it kills something for pants (maybe the daemon?).

Pants version 2.15.0

OS Ubuntu 20.04

Additional info Here are the logs when the command throws the error.

11:19:28.90 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list --changed-since=master`
11:19:43.80 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list --changed-since=master`
11:19:44.49 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list astranis-python/astranis/ground/service/angle_collector:sources astranis-python/astranis/ground/service/angle_collector/client_server_test.py:tests astranis-python/astranis/service/handler.py astranis-python/astranis/ground/service/angle_collector:client astranis-python/astranis/ground/service/angle_collector/client.py:sources`
11:19:44.98 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list astranis-python/astranis/ground/service/angle_collector:sources astranis-python/astranis/ground/service/angle_collector/client_server_test.py:tests astranis-python/astranis/service/handler.py astranis-python/astranis/ground/service/angle_collector:client astranis-python/astranis/ground/service/angle_collector/client.py:sources`
.sleep(self.JOIN_TIMEOUT_SECONDS)
  File "/home/nathanael/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 109, in handle_sigterm
    raise self.SignalHandledNonLocalExit(signum, "SIGTERM")
  File "/home/nathanael/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 94, in __init__
    self.traceback_lines = traceback.format_stack()

11:19:45.89 [WARN] Executor was not shut down explicitly.
11:19:45.89 [ERROR] panic at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/blocking/shutdown.rs:51
11:19:45.89 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
11:19:45.89 [ERROR] panic at 'A Tokio 1.x context was found, but it is being shutdown.', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/time/driver/entry.rs:558
11:19:45.89 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
11:19:45.89 [ERROR] panic at 'panic in a function that cannot unwind', library/core/src/panicking.rs:106
11:19:45.89 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
jsirois commented 1 year ago

@engnatha can you try with 2.16.0.dev7 or higher? (Preferably 2.16.0rc1). This sounds like a bug @stuhood fixed.

fengwei-tian commented 1 year ago

We still found this issue in the 2.17.0rc1_py39 Here are the logs: timestamp: 2023-07-08T05:13:43.183791 process title: pantsd [/var/lib/buildkite-agent/builds/buildkite-agent-rippling-main-tests-batches-green-stack-i-01ab64de5614ee8b9-1/rippling/rippling-main-pr-tests] sys.argv: ['/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/bin/pants', '--pants-bin-name=./pants', '--pants-version=2.17.0rc1', '--spec-files=changed-files.txt', 'test-selection-method', '--output-file=selection-method.txt'] pid: 32055 Signal 15 (SIGTERM) was raised. Exiting with failure. File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/bin/pants", line 8, in sys.exit(main()) File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main PantsLoader.main() File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 106, in main cls.run_alternate_entrypoint(entrypoint) File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 84, in run_alternate_entrypoint entrypoint_fn() File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/pantsd/pants_daemon.py", line 238, in launch_new_pantsd_instance daemon.run_sync() File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/pantsd/pants_daemon.py", line 211, in run_sync time.sleep(self.JOIN_TIMEOUT_SECONDS) File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 109, in handle_sigterm raise self.SignalHandledNonLocalExit(signum, "SIGTERM") File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 94, in init self.traceback_lines = traceback.format_stack()

Fatal Python error: Aborted

Thread 0x00007fbc3b77f680 (most recent call first):