search

facebookresearch / torchbeast

A PyTorch Platform for Distributed RL
Apache License 2.0
734 stars 113 forks source link

Don't require Python 3.7 EXACT in CMakeLists.txt #29

Closed adyomin closed 3 years ago

adyomin commented 3 years ago

Based on my unsuccsessful attempt to build PolyBeast in a Python 3.8 environment, I understand that something explicitly requires 3.7. If my understanding is correct, could you please update PolyBeast installation instructions accordingly?

facebook-github-bot commented 3 years ago

Hi @adyomin!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

facebook-github-bot commented 3 years ago

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

heiner commented 3 years ago

Hey Andrei,

Thanks for your PR, and sorry that you had issues building TorchBeast (let's just say you're not alone in that).

It's true that we used Python 3.7 when we build this agent, but I would have hoped later versions of Python work too. Could you share how it fails for these versions?

adyomin commented 3 years ago

Just did a clean test from scratch (except for repo and sub-modules cloning):

adyomin@DLW ~/s/torchbeast (master)> conda create --name nle_38 python=3.8                                                                                          (nle_37) 
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/adyomin/miniconda3/envs/nle_38

  added / updated specs:
    - python=3.8

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.5.25-h06a4308_1
  certifi            pkgs/main/linux-64::certifi-2021.5.30-py38h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1k-h27cfd23_0
  pip                pkgs/main/linux-64::pip-21.1.2-py38h06a4308_0
  python             pkgs/main/linux-64::python-3.8.10-h12debd9_8
  readline           pkgs/main/linux-64::readline-8.1-h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-52.0.0-py38h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.35.4-hdfb4753_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  wheel              pkgs/main/noarch::wheel-0.36.2-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate nle_38
#
# To deactivate an active environment, use
#
#     $ conda deactivate

adyomin@DLW ~/s/torchbeast (master)> conda activate nle_38                                                                                                          (nle_37) 
adyomin@DLW ~/s/torchbeast (master)> conda install pytorch cudatoolkit=11.1 -c pytorch -c nvidia                                                                    (nle_38) 
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/adyomin/miniconda3/envs/nle_38

  added / updated specs:
    - cudatoolkit=11.1
    - pytorch

The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  cudatoolkit        nvidia/linux-64::cudatoolkit-11.1.74-h6bb024c_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.2.0-h06a4308_610
  libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
  mkl                pkgs/main/linux-64::mkl-2021.2.0-h06a4308_296
  ninja              pkgs/main/linux-64::ninja-1.10.2-hff7bd54_1
  pytorch            pytorch/linux-64::pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0
  typing_extensions  pkgs/main/noarch::typing_extensions-3.7.4.3-pyha847dfd_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: - By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done
adyomin@DLW ~/s/torchbeast (master)>  

adyomin@DLW ~/s/torchbeast (master)> pip install -r requirements.txt                                                                                                (nle_38) 
Collecting gym[atari]>=0.14.0
  Using cached gym-0.18.3-py3-none-any.whl
Collecting atari-py==0.2.5
  Using cached atari_py-0.2.5-cp38-cp38-linux_x86_64.whl
Collecting gitpython>=2.1
  Using cached GitPython-3.1.18-py3-none-any.whl (170 kB)
Collecting opencv-python
  Using cached opencv_python-4.5.2.54-cp38-cp38-manylinux2014_x86_64.whl (51.0 MB)
Collecting flake8
  Using cached flake8-3.9.2-py2.py3-none-any.whl (73 kB)
Collecting black
  Using cached black-21.6b0-py3-none-any.whl (140 kB)
Collecting pre-commit
  Using cached pre_commit-2.13.0-py2.py3-none-any.whl (190 kB)
Collecting numpy
  Using cached numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.7-py3-none-any.whl (63 kB)
Collecting smmap<5,>=3.0.1
  Using cached smmap-4.0.0-py2.py3-none-any.whl (24 kB)
Collecting cloudpickle<1.7.0,>=1.2.0
  Using cached cloudpickle-1.6.0-py3-none-any.whl (23 kB)
Collecting Pillow<=8.2.0
  Using cached Pillow-8.2.0-cp38-cp38-manylinux1_x86_64.whl (3.0 MB)
Collecting pyglet<=1.5.15,>=1.4.0
  Using cached pyglet-1.5.15-py3-none-any.whl (1.1 MB)
Collecting scipy
  Using cached scipy-1.6.3-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
Collecting pycodestyle<2.8.0,>=2.7.0
  Using cached pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB)
Collecting mccabe<0.7.0,>=0.6.0
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.4.0,>=2.3.0
  Using cached pyflakes-2.3.1-py2.py3-none-any.whl (68 kB)
Collecting pathspec<1,>=0.8.1
  Using cached pathspec-0.8.1-py2.py3-none-any.whl (28 kB)
Collecting toml>=0.10.1
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting click>=7.1.2
  Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting appdirs
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting regex>=2020.1.8
  Using cached regex-2021.4.4-cp38-cp38-manylinux2014_x86_64.whl (733 kB)
Collecting pyyaml>=5.1
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting nodeenv>=0.11.1
  Using cached nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
Collecting cfgv>=2.0.0
  Using cached cfgv-3.3.0-py2.py3-none-any.whl (7.3 kB)
Collecting virtualenv>=20.0.8
  Using cached virtualenv-20.4.7-py2.py3-none-any.whl (7.2 MB)
Collecting identify>=1.0.0
  Using cached identify-2.2.10-py2.py3-none-any.whl (98 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.2-py2.py3-none-any.whl (338 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Installing collected packages: numpy, smmap, six, scipy, pyglet, Pillow, filelock, distlib, cloudpickle, appdirs, virtualenv, toml, regex, pyyaml, pyflakes, pycodestyle, pathspec, opencv-python, nodeenv, mypy-extensions, mccabe, identify, gym, gitdb, click, cfgv, atari-py, pre-commit, gitpython, flake8, black
Successfully installed Pillow-8.2.0 appdirs-1.4.4 atari-py-0.2.5 black-21.6b0 cfgv-3.3.0 click-8.0.1 cloudpickle-1.6.0 distlib-0.3.2 filelock-3.0.12 flake8-3.9.2 gitdb-4.0.7 gitpython-3.1.18 gym-0.18.3 identify-2.2.10 mccabe-0.6.1 mypy-extensions-0.4.3 nodeenv-1.6.0 numpy-1.20.3 opencv-python-4.5.2.54 pathspec-0.8.1 pre-commit-2.13.0 pycodestyle-2.7.0 pyflakes-2.3.1 pyglet-1.5.15 pyyaml-5.4.1 regex-2021.4.4 scipy-1.6.3 six-1.16.0 smmap-4.0.0 toml-0.10.2 virtualenv-20.4.7

adyomin@DLW ~/s/torchbeast (master)> pip install nest/                                                                                                              (nle_38) 
Processing ./nest
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Collecting pybind11>=2.3
  Using cached pybind11-2.6.2-py2.py3-none-any.whl (191 kB)
Building wheels for collected packages: nest
  Building wheel for nest (setup.py) ... done
  Created wheel for nest: filename=nest-0.0.3-cp38-cp38-linux_x86_64.whl size=1359981 sha256=51687f8822780c0a48902e3929ecb001da9095838828c9142f849901d17343d1
  Stored in directory: /tmp/pip-ephem-wheel-cache-isfh3ukc/wheels/3b/d0/ff/525487a4ac7f26e39949ecdbe566c3ac2a9fbc6b7eabd702a9
Successfully built nest
Installing collected packages: pybind11, nest
Successfully installed nest-0.0.3 pybind11-2.6.2

adyomin@DLW ~/s/torchbeast (master)> python setup.py install                                                                                                        (nle_38) 
running install
running bdist_egg
running egg_info
writing libtorchbeast.egg-info/PKG-INFO
writing dependency_links to libtorchbeast.egg-info/dependency_links.txt
writing requirements to libtorchbeast.egg-info/requires.txt
writing top-level names to libtorchbeast.egg-info/top_level.txt
reading manifest file 'libtorchbeast.egg-info/SOURCES.txt'
writing manifest file 'libtorchbeast.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- Could NOT find Python3 (missing: Python3_EXECUTABLE Python3_INCLUDE_DIRS Python3_LIBRARIES Python3_NumPy_INCLUDE_DIRS Interpreter Development NumPy Development.Module Development.Embed) (Required is exact version "3.7")
    Reason given by package: 
        Interpreter: Wrong version for the interpreter "/home/adyomin/miniconda3/envs/nle_38/bin/python3"

-- pybind11 v2.6.2 
-- Found PythonInterp: /home/adyomin/miniconda3/envs/nle_38/bin/python (found version "3.8.10") 
-- Found PythonLibs: /home/adyomin/miniconda3/envs/nle_38/lib
-- 
-- 3.14.0.0
-- Caffe2: CUDA detected: 11.3
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.3
-- Found cuDNN: v8.2.1  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 1ea278b5
-- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
CMake Warning at /home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:51 (find_package)

-- Configuring done
CMake Error at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Target "_C" links to target "Python3::NumPy" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

CMake Error at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Target "_C" links to target "Python3::NumPy" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

CMake Warning at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Cannot generate a safe runtime search path for target _C because there is a
  cycle in the constraint graph:

    dir 0 is [/home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/lib]
    dir 1 is [/usr/local/cuda/lib64/stubs]
    dir 2 is [/usr/local/cuda/lib64]
      dir 3 must precede it due to runtime library [libnvToolsExt.so.1]
    dir 3 is [/home/adyomin/miniconda3/envs/nle/lib]
      dir 2 must precede it due to runtime library [libcudart.so.11.0]
    dir 4 is [/home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages/torch/lib]

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

Part with the 3.7 requirement:

running install
running bdist_egg
running egg_info
writing libtorchbeast.egg-info/PKG-INFO
writing dependency_links to libtorchbeast.egg-info/dependency_links.txt
writing requirements to libtorchbeast.egg-info/requires.txt
writing top-level names to libtorchbeast.egg-info/top_level.txt
reading manifest file 'libtorchbeast.egg-info/SOURCES.txt'
writing manifest file 'libtorchbeast.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- Could NOT find Python3 (missing: Python3_EXECUTABLE Python3_INCLUDE_DIRS Python3_LIBRARIES Python3_NumPy_INCLUDE_DIRS Interpreter Development NumPy Development.Module Development.Embed) (Required is exact version "3.7")
    Reason given by package: 
        Interpreter: Wrong version for the interpreter "/home/adyomin/miniconda3/envs/nle_38/bin/python3"
heiner commented 3 years ago

Hey Andrei,

Thanks for the quick response.

Could you move this output into a new issue, and also include the output of this environment collection script

You can get the script and run it with:

wget https://raw.githubusercontent.com/facebookresearch/nle/master/nle/scripts/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
adyomin commented 3 years ago

Done

heiner commented 3 years ago

Thanks, merged.