search

giampaolo / psutil

Cross-platform lib for process and system monitoring in Python
BSD 3-Clause "New" or "Revised" License
10.08k stars 1.37k forks source link

[Windows] `pid_exists()` and `Process()` disagree on whether a pid exists when ERROR_ACCESS_DENIED #2359

Closed tapple closed 18 hours ago

tapple commented 5 months ago

Summary

Description

I don't know how I got my system into this state, but these 2 functions disagree on whether this PID exists:

(rk39) PS C:\Users\mfulmer\cabbage\standalone> python
Python 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.pid_exists(17320)
True
>>> psutil.Process(17320)
Traceback (most recent call last):
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\_pswindows.py", line 730, in wrapper
    return fun(self, *args, **kwargs)
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\_pswindows.py", line 995, in create_time
    return self._proc_info()[pinfo_map['create_time']]
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\_common.py", line 497, in wrapper
    raise raise_from(err, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\_common.py", line 495, in wrapper
    return fun(self)
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\_pswindows.py", line 790, in _proc_info
    ret = cext.proc_info(self.pid)
ProcessLookupError: [Errno 3] assume no such process (originated from NtQuerySystemInformation (no PID found))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\__init__.py", line 359, in _init
    self.create_time()
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\__init__.py", line 752, in create_time
    self._create_time = self._proc.create_time()
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\_pswindows.py", line 732, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.NoSuchProcess: process no longer exists (pid=17320)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\__init__.py", line 323, in __init__
    self._init(pid)
  File "C:\Users\mfulmer\Miniconda3\envs\rk39\lib\site-packages\psutil\__init__.py", line 372, in _init
    raise NoSuchProcess(pid, msg=msg)
psutil.NoSuchProcess: process PID not found (pid=17320)

I stepped thru both in a debugger. I couldn't step thru pid_exists, since it's directly from c, but psutil.Process() is failing here:

>>> import psutil._psutil_windows
>>> psutil._psutil_windows.proc_times(17320)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [WinError 5] Access is denied: '(originated from OpenProcess)'

Searching for 17320 in task manager gives no results, so I think the PID does not, in fact, exist

tapple commented 5 months ago

I did some more testing. It looks like there's a lot more than pid 17320 where there's a mismatch:

>>> import psutil
>>> import psutil._psutil_windows
>>> pidinfo = []
>>> for i in range(30000):
...     exists = psutil.pid_exists(i)
...     try:
...         proc = psutil.Process(i)
...     except Exception as e:
...         proc = e
...     try:
...         times = psutil._psutil_windows.proc_times(i)
...     except Exception as e:
...         times = e
...     pidinfo.append((exists, proc, times))
>>> mismatches = [exists != isinstance(proc, psutil.Process) for exists, proc, times in pidinfo]
>>> mismatched_pids=[i for i, mismatched in enumerate(mismatches) if mismatched]
>>> mismatched_info=[(i, ) + pidinfo[i] for i in mismatched_pids]
>>> running_pids = [i for i, (exists, proc, times) in enumerate(pidinfo) if exists]
>>> len(running_pids)
2801
>>> len(mismatched_info)
1923

>>> running_pids[:10]
[0, 4, 5, 6, 7, 68, 69, 70, 71, 108]
>>> mismatched_info[:10]
[
    (5, True, psutil.NoSuchProcess(pid=5, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (6, True, psutil.NoSuchProcess(pid=6, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (7, True, psutil.NoSuchProcess(pid=7, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (68, True, psutil.NoSuchProcess(pid=68, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (69, True, psutil.NoSuchProcess(pid=69, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (70, True, psutil.NoSuchProcess(pid=70, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (71, True, psutil.NoSuchProcess(pid=71, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (109, True, psutil.NoSuchProcess(pid=109, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (110, True, psutil.NoSuchProcess(pid=110, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (111, True, psutil.NoSuchProcess(pid=111, msg="process PID not found"), PermissionError(13, "Access is denied")),
]

>>> running_pids[-10:]
[28498, 28499, 28520, 28521, 28522, 28523, 28592, 28593, 28594, 28595]
>>> mismatched_info[-10:]
[
    (28494, True, psutil.NoSuchProcess(pid=28494, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28495, True, psutil.NoSuchProcess(pid=28495, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28496, True, psutil.NoSuchProcess(pid=28496, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28497, True, psutil.NoSuchProcess(pid=28497, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28498, True, psutil.NoSuchProcess(pid=28498, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28499, True, psutil.NoSuchProcess(pid=28499, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28592, True, psutil.NoSuchProcess(pid=28592, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28593, True, psutil.NoSuchProcess(pid=28593, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28594, True, psutil.NoSuchProcess(pid=28594, msg="process PID not found"), PermissionError(13, "Access is denied")),
    (28595, True, psutil.NoSuchProcess(pid=28595, msg="process PID not found"), PermissionError(13, "Access is denied")),
]

>>> pidinfo[17320]
(True, psutil.NoSuchProcess(pid=17320, msg="process PID not found"), PermissionError(13, "Access is denied"))

I checked the first 10 and last 10 mismatched pid's in task manager (I made sure to run it as Administrator). None were actually running. It seems that psutil.Process() is right when it doesn't match psutil.pid_exists()

I know that 17320 is a process that used to be running, but no longer is. I wonder if that's true of the rest of these. I'm pretty sure 17320 was a python process in vscode debugger that I force-killed by stopping the debugger

tapple commented 5 months ago

Since this is a permission denied error, I thought I'd see what happens if I actually used multiple users (I ran these tests on windows 10, not 11)

on tapple account (administrator):

(psutil-venv) C:\Users\tapple>python
Python 3.11.0 (main, Oct 24 2022, 18:13:38) [MSC v.1933 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> p = psutil.Process()
>>> p
psutil.Process(pid=3592, name='python.exe', status='running', started='08:27:57')

on testuser account (not administrator):

(psutil-venv) C:\Users\testuser>python
Python 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> p = psutil.Process()
>>> p
psutil.Process(pid=13180, name='python3.12.exe', status='running', started='08:30:05')
>>> psutil.pid_exists(3592)
True
>>> other = psutil.Process(3592)
>>> other
psutil.Process(pid=3592, name='python.exe', status='running', started='08:27:57')
>>> other.username()
Traceback (most recent call last):
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 730, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 983, in username
    domain, user = cext.proc_username(self.pid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [WinError 5] Access is denied: '(originated from OpenProcess)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\__init__.py", line 744, in username
    return self._proc.username()
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 732, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.AccessDenied: (pid=3592, name='python.exe')
>>> other.kill()
Traceback (most recent call last):
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 730, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 917, in kill
    return cext.proc_kill(self.pid)
           ^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [WinError 5] Access is denied: '(originated from OpenProcess)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\__init__.py", line 1304, in kill
    self._proc.kill()
  File "C:\Users\testuser\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 732, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.AccessDenied: (pid=3592, name='python.exe')

back on tapple account (administrator):

>>> psutil.pid_exists(13180)
True
>>> other = psutil.Process(13180)
>>> other
psutil.Process(pid=13180, name='python3.12.exe', status='running', started='08:30:05')
>>> other.username()
Traceback (most recent call last):
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 730, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 983, in username
    domain, user = cext.proc_username(self.pid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [WinError 5] Access is denied: '(originated from OpenProcess)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\__init__.py", line 744, in username
    return self._proc.username()
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 732, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.AccessDenied: (pid=13180, name='python3.12.exe')
>>> other.kill()
Traceback (most recent call last):
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 730, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 917, in kill
    return cext.proc_kill(self.pid)
           ^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [WinError 5] Access is denied: '(originated from OpenProcess)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\__init__.py", line 1304, in kill
    self._proc.kill()
  File "C:\Users\tapple\psutil-venv\Lib\site-packages\psutil\_pswindows.py", line 732, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.AccessDenied: (pid=13180, name='python3.12.exe')

So, psutil is able to get basic info about a process of another user without permission denied (including time), but not more detailed info.

In conclusion I think the correct behavior is: If there is permission denied getting basic process info, the process really does not exist. So:

tapple commented 5 months ago

There's a similar bug for linux, #1910. In that bug, it's the opposite: pid_exists() is correct and Process() is incorrect. So, the only sure-fire way to check that a process exists, in spite of these two bugs is:

proc = None
try:
    # check both pid_exists and Process. There are at least
    # 2 issues about them disagreeing in different directions.
    # See issues #1910 and #2359
    if psutil.pid_exists(pid):
        proc = psutil.Process(pid)
except (ProcessLookupError, psutil.NoSuchProcess):
    pass

if proc is not None:
    ...
giampaolo commented 3 months ago

This is a reproducible test case that works on Linux and FreeBSD but on Windows, showing how pid_exists() on Windows is indeed wrong:

    def test_pid_exists_and_process_agree(self):
        def is_linux_tid(pid):
            try:
                f = open("/proc/%s/status" % pid, "rb")
            except FileNotFoundError:
                return False
            else:
                with f:
                    for line in f:
                        if line.startswith(b"Tgid:"):
                            tgid = int(line.split()[1])
                            # If tgid and pid are different then we're
                            # dealing with a process TID.
                            return tgid != pid
                    raise ValueError("'Tgid' line not found in %s" % path)

        for pid in range(1, 30000):
            if LINUX and is_linux_tid(pid):
                # On Linux a TID (thread ID) can be passed to the
                # Process class and is querable like a PID (process
                # ID). Skip it.
                continue
            exists = psutil.pid_exists(pid)
            if exists:
                psutil.Process(pid)
            else:
                with self.assertRaises(psutil.NoSuchProcess):
                    psutil.Process(pid)
giampaolo commented 3 months ago

Problem appears to be here: https://github.com/giampaolo/psutil/blob/034a1a6996d4ff5116fc45a9c5ed8477d0d73d37/psutil/arch/windows/proc_utils.c#L176-L178.

I've tried exercising this with PID 5 (which does not exist). The assumption in the code is that "ERROR_ACCESS_DENIED means there's a process to deny access to (hence it exists)", but this assumption (which is true on UNIX) appears to be wrong.

giampaolo commented 3 months ago

PR: https://github.com/giampaolo/psutil/pull/2394.

giampaolo commented 18 hours ago

Closing as fixed by https://github.com/giampaolo/psutil/pull/2394/.