search

giampaolo / psutil

Cross-platform lib for process and system monitoring in Python
BSD 3-Clause "New" or "Revised" License
10.22k stars 1.38k forks source link

psutil fails (segfault) on Mac OS in the free-threaded (no GIL) version of Python 3.13 #2427

Closed pkoning2 closed 3 months ago

pkoning2 commented 3 months ago

Summary

Description

Transfered here from https://github.com/python/cpython/issues/120717

I used pip to install psutil on python 3.13 0b2, both the regular and the "t" version. The regular one works; the t one fails at import psutil with a segfault.

The Python teams comments: "Any extensions that uses more than Python would need to adapt to free-threading if they want to onboard. It's impossible for CPython to ensure all the C extensions work without GIL. So this should be an issue for psutil ."

giampaolo commented 3 months ago

I don't know enough about free-thread cPython to give any guidance. Some work in this sense was done in https://github.com/giampaolo/psutil/issues/2401. @colesbury any idea?

KRRT7 commented 3 months ago

happens on windows as well.

giampaolo commented 3 months ago

Can you guys run:

python3 -c "import faulthandler; faulthandler.enable(); import psutil"

...and paste the output?

KRRT7 commented 3 months ago 
Administrator in ~\Documents\tests via 🐍 v3.13.0b1 (313t-venv)
❯ python -c "import faulthandler; faulthandler.enable(); import psutil"
Windows fatal exception: access violation

Current thread 0x00002388 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1315 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "C:\Users\NuitkaDevOps\Documents\cmbos\313t-venv\Lib\site-packages\psutil\_pswindows.py", line 35 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1021 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  File "C:\Users\NuitkaDevOps\Documents\cmbos\313t-venv\Lib\site-packages\psutil\__init__.py", line 110 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1021 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<string>", line 1 in <module>
pkoning2 commented 3 months ago

I get this, on Mac OS:

pkoning:foo pkoning$ PYTHONFAULTHANDLER=1 python3.13t -c "import psutil"
Fatal Python error: Segmentation fault

Current thread 0x0000000104750580 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1316 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  File "/Library/Frameworks/PythonT.framework/Versions/3.13/lib/python3.13/site-packages/psutil/_psposix.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  File "/Library/Frameworks/PythonT.framework/Versions/3.13/lib/python3.13/site-packages/psutil/_psosx.py", line 13 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  File "/Library/Frameworks/PythonT.framework/Versions/3.13/lib/python3.13/site-packages/psutil/__init__.py", line 124 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<string>", line 1 in <module>
Segmentation fault: 11

This is with the published release (from pip install). Should we try it with the current latest code from github?

colesbury commented 3 months ago

Sorry for the ping ponging of issues between projects. I think this is mostly not specific to psutil, but I'm not entirely sure yet.

I'd like to focus on macOS first, because I think it'll be easier to get it into a good state.

@pkoning2, how did you get Python, pip, and psutil installed?

Here are some issues I noticed on macOS:

Here's what I did to successfully install psutil:

There's still the issue that importing psutil enables the GIL:

<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'psutil._psutil_osx', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

That can be fixed in this project, but the other issues are more CPython related.

pkoning2 commented 3 months ago

Yes, what I did roughly follows the steps you describe. I didn't realize that pip was out of date and this is the first time I've run into a situation where it actually matters.

So yes, now you have the GIL lock complaint. I ran python3.13t with that flag to suppress the warning, and psutil seems to work fine. At least for net_if_addrs() which is the only function I have used so far. So perhaps all that is needed is to supply the flag that says to Python that you're compatible with the free-threading flavor?

I noticed the help mentions net_connections(); that fails on 3.13 (either flavor) with an "operation not permitted" exception, but it seems to work on 3.12.

colesbury commented 3 months ago

I'll work on a patch to suppress the GIL warnings and behavior in the free-threaded build, as well as a few small thread-safety issues on some OS's if certain psutil functions are used from multiple threads concurrently.