pythonprofilers / memory_profiler

Monitor Memory usage of Python code
http://pypi.python.org/pypi/memory_profiler
Other
4.39k stars 380 forks source link

psutil.NoSuchProcess error #315

Open VolodyaCO opened 3 years ago

VolodyaCO commented 3 years ago

I am using memory profiler version 0.58.0 to do some profiling on a function that I run on daily on a Digital Ocean droplet. It has worked for months. Sudenly, today, the script raised an error. This is the "script":

try:
    mem_usage = memory_usage((main, (today,), {}), interval=1.0)
    print('avg memory usage was', np.mean(mem_usage))
    print('peaks of memory usage were', np.sort(mem_usage)[-10:]) 
except Exception:
    print(traceback.format_exc())
    shutil.rmtree(todaypath)

The main function basically uses numpy, pandas, scipy, and pymc3.

And this is the error. As you see, no traceback is related to any parts of my code

Process MemTimer-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/psutil/_common.py", line 447, in wrapper
    ret = self._cache[fun]
AttributeError: _cache
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1576, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/psutil/_common.py", line 450, in wrapper
    return fun(self)
  File "/usr/local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1618, in _parse_stat_file
    with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f:
  File "/usr/local/lib/python3.7/site-packages/psutil/_common.py", line 711, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/5059/stat'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/psutil/__init__.py", line 354, in _init
    self.create_time()
  File "/usr/local/lib/python3.7/site-packages/psutil/__init__.py", line 710, in create_time
    self._create_time = self._proc.create_time()
  File "/usr/local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1576, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1788, in create_time
    ctime = float(self._parse_stat_file()['create_time'])
  File "/usr/local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1583, in wrapper
    raise NoSuchProcess(self.pid, self._name)
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5059)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/site-packages/memory_profiler.py", line 217, in run
    include_children=self.include_children,)
  File "/usr/local/lib/python3.7/site-packages/memory_profiler.py", line 184, in _get_memory
    return tools[backend]()
  File "/usr/local/lib/python3.7/site-packages/memory_profiler.py", line 128, in ps_util_tool
    process = psutil.Process(pid)
  File "/usr/local/lib/python3.7/site-packages/psutil/__init__.py", line 326, in __init__
    self._init(pid)
  File "/usr/local/lib/python3.7/site-packages/psutil/__init__.py", line 367, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 5059

As a matter of fact, when I run the code without the memory_usage it runs correctly as expected.