AttributeError: 'Process' object has no attribute 'cpu_affinity'

[billtubbs]

Has anyone else had this error with psutil?

Traceback (most recent call last):
  File "examples/atari_dqn_async_cpu.py", line 70, in <module>
    run_ID=args.run_ID,
  File "examples/atari_dqn_async_cpu.py", line 59, in build_and_train
    runner.train()
  File "/Users/billtubbs/rlpyt/rlpyt/runners/async_rl.py", line 40, in train
    throttle_itr, delta_throttle_itr = self.startup()
  File "/Users/billtubbs/rlpyt/rlpyt/runners/async_rl.py", line 108, in startup
    throttle_itr, delta_throttle_itr = self.optim_startup()
  File "/Users/billtubbs/rlpyt/rlpyt/runners/async_rl.py", line 115, in optim_startup
    p.cpu_affinity(main_affinity["cpus"])
AttributeError: 'Process' object has no attribute 'cpu_affinity'

Seems to be a problem with my psutil version on Mac OS. My current version is 5.6.3.

This illustrates the root of the problem:

(rlpyt) Bills-Mac-mini-2:rlpyt billtubbs$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.__version__
'5.6.3'
>>> import psutil
>>> p = psutil.Process()
>>> p.cpu_affinity
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Process' object has no attribute 'cpu_affinity'

Full installation and execution details:

(base) Bills-Mac-mini-2:rlpyt billtubbs$ ls
LICENSE         linux_cpu.yml       rlpyt.egg-info
README.md       linux_cuda10.yml    scratch
data            linux_cuda9.yml     setup.py
examples        macos_cpu.yml
images          rlpyt
(base) Bills-Mac-mini-2:rlpyt billtubbs$ conda env create -f macos_cpu.yml
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
pytorch-1.3.0        | 32.6 MB   | ################################### | 100% 
setuptools-41.4.0    | 644 KB    | ################################### | 100% 
wheel-0.33.6         | 40 KB     | ################################### | 100% 
sqlite-3.30.0        | 2.4 MB    | ################################### | 100% 
numpy-base-1.17.2    | 5.0 MB    | ################################### | 100% 
numpy-1.17.2         | 4 KB      | ################################### | 100% 
pip-19.2.3           | 1.9 MB    | ################################### | 100% 
mkl_random-1.1.0     | 320 KB    | ################################### | 100% 
mkl-service-2.3.0    | 187 KB    | ################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Ran pip subprocess with arguments:
['/anaconda3/envs/rlpyt/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/billtubbs/rlpyt/condaenv.m7_60k9g.requirements.txt']
Pip subprocess output:
Collecting atari-py (from -r /Users/billtubbs/rlpyt/condaenv.m7_60k9g.requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/f9/bb/b01e780ac5abc41f41b37faeaa005fcdabffe37713d943f86190305c8294/atari_py-0.2.6-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting pyprind (from -r /Users/billtubbs/rlpyt/condaenv.m7_60k9g.requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/1e/30/e76fb0c45da8aef49ea8d2a90d4e7a6877b45894c25f12fb961f009a891e/PyPrind-2.11.2-py3-none-any.whl
Requirement already satisfied, skipping upgrade: numpy in /anaconda3/envs/rlpyt/lib/python3.7/site-packages (from atari-py->-r /Users/billtubbs/rlpyt/condaenv.m7_60k9g.requirements.txt (line 1)) (1.17.2)
Requirement already satisfied, skipping upgrade: six in /anaconda3/envs/rlpyt/lib/python3.7/site-packages (from atari-py->-r /Users/billtubbs/rlpyt/condaenv.m7_60k9g.requirements.txt (line 1)) (1.12.0)
Installing collected packages: atari-py, pyprind
Successfully installed atari-py-0.2.6 pyprind-2.11.2

#
# To activate this environment, use
#
#     $ conda activate rlpyt
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) Bills-Mac-mini-2:rlpyt billtubbs$ source activate rlpyt
(rlpyt) Bills-Mac-mini-2:rlpyt billtubbs$ python examples/atari_dqn_async_cpu.py
Traceback (most recent call last):
  File "examples/atari_dqn_async_cpu.py", line 7, in <module>
    from rlpyt.utils.launching.affinity import make_affinity
ModuleNotFoundError: No module named 'rlpyt'
(rlpyt) Bills-Mac-mini-2:rlpyt billtubbs$ pip install -e .
Obtaining file:///Users/billtubbs/rlpyt
Installing collected packages: rlpyt
  Running setup.py develop for rlpyt
Successfully installed rlpyt
(rlpyt) Bills-Mac-mini-2:rlpyt billtubbs$ python examples/atari_dqn_async_cpu.py
cat: /proc/cpuinfo: No such file or directory
logger_context received log_dir outside of /Users/billtubbs/rlpyt/data: prepending by /Users/billtubbs/rlpyt/data/local/<yyyymmdd>/
using seed 9387
2019-10-12 11:07:28.712698  | async_dqn_pong_0 Running 50000 sampler iterations.
2019-10-12 11:07:29.073120  | async_dqn_pong_0 Frame-based buffer using 4-frame sequences.
2019-10-12 11:07:46.956154  | async_dqn_pong_0 Agent setting min/max epsilon itrs: 250, 25000
Traceback (most recent call last):
  File "examples/atari_dqn_async_cpu.py", line 70, in <module>
    run_ID=args.run_ID,
  File "examples/atari_dqn_async_cpu.py", line 59, in build_and_train
    runner.train()
  File "/Users/billtubbs/rlpyt/rlpyt/runners/async_rl.py", line 40, in train
    throttle_itr, delta_throttle_itr = self.startup()
  File "/Users/billtubbs/rlpyt/rlpyt/runners/async_rl.py", line 108, in startup
    throttle_itr, delta_throttle_itr = self.optim_startup()
  File "/Users/billtubbs/rlpyt/rlpyt/runners/async_rl.py", line 115, in optim_startup
    p.cpu_affinity(main_affinity["cpus"])
AttributeError: 'Process' object has no attribute 'cpu_affinity'

Or, maybe I shouldn't be trying to run this example on Mac OS with CPUs?

[billtubbs]

I found this project that had this issue and resolved it:

• crossbar: https://github.com/crossbario/crossbar/issues/1333

[billtubbs]

In the psutil documentation for cpu_affinity it says:

Availability: Linux, Windows, FreeBSD

[astooke]

Hi, thanks for the question!

Short answer: to run on MacOS, comment out the lines with cpu_affinity.

Longer answer: Yes, my understanding is that the cpu_affinity functionality is not available in MacOS. Some places in the code (like the sampler parallel worker), there's a try statement to protect against this. I must've left this out of the async code (developed a little later). Also the async code only really makes sense with GPUs, which I figured would be on Linux and not MacOS (but maybe so?). In general in MacOS I don't think you can control the cpu affinity of processes.

Why affinity: In some experiments I found sampling to run faster when assigning the CPU of each processes, so they stay on their own hardware lanes and the system doesn't get bogged down with the OS's cpu scheduler shifting processes all around. But it's not a hard and fast rule, just something to try for your own program. :)

[billtubbs]

Thanks. This makes sense. Maybe I shouldn't be trying to run this example. What's a good example script to get me started on Mac OS X with CPUs? (I'm just trying out this package for the first time and will transition to a linux-GPU machine once I figured it out).

[billtubbs]

Looks like examples/example_1.py seems to be the best place to start. This is working now. Thanks.