search

facebookresearch / torchbeast

A PyTorch Platform for Distributed RL
Apache License 2.0
734 stars 113 forks source link

PolyBeast build fails with Python 3.8 #30

Closed adyomin closed 3 years ago

adyomin commented 3 years ago

I tired following the NetHack challenge baseline setup instructions using Python 3.8 as suggested. I could not build PolyBeast.

Steps to reproduce the issue (except for repo and sub-modules cloning):

adyomin@DLW ~/s/torchbeast (master)> conda create --name nle_38 python=3.8                                                                                          (nle_37) 
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/adyomin/miniconda3/envs/nle_38

  added / updated specs:
    - python=3.8

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.5.25-h06a4308_1
  certifi            pkgs/main/linux-64::certifi-2021.5.30-py38h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1k-h27cfd23_0
  pip                pkgs/main/linux-64::pip-21.1.2-py38h06a4308_0
  python             pkgs/main/linux-64::python-3.8.10-h12debd9_8
  readline           pkgs/main/linux-64::readline-8.1-h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-52.0.0-py38h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.35.4-hdfb4753_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  wheel              pkgs/main/noarch::wheel-0.36.2-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate nle_38
#
# To deactivate an active environment, use
#
#     $ conda deactivate

adyomin@DLW ~/s/torchbeast (master)> conda activate nle_38                                                                                                          (nle_37) 
adyomin@DLW ~/s/torchbeast (master)> conda install pytorch cudatoolkit=11.1 -c pytorch -c nvidia                                                                    (nle_38) 
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/adyomin/miniconda3/envs/nle_38

  added / updated specs:
    - cudatoolkit=11.1
    - pytorch

The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  cudatoolkit        nvidia/linux-64::cudatoolkit-11.1.74-h6bb024c_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.2.0-h06a4308_610
  libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
  mkl                pkgs/main/linux-64::mkl-2021.2.0-h06a4308_296
  ninja              pkgs/main/linux-64::ninja-1.10.2-hff7bd54_1
  pytorch            pytorch/linux-64::pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0
  typing_extensions  pkgs/main/noarch::typing_extensions-3.7.4.3-pyha847dfd_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: - By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done
adyomin@DLW ~/s/torchbeast (master)>  

adyomin@DLW ~/s/torchbeast (master)> pip install -r requirements.txt                                                                                                (nle_38) 
Collecting gym[atari]>=0.14.0
  Using cached gym-0.18.3-py3-none-any.whl
Collecting atari-py==0.2.5
  Using cached atari_py-0.2.5-cp38-cp38-linux_x86_64.whl
Collecting gitpython>=2.1
  Using cached GitPython-3.1.18-py3-none-any.whl (170 kB)
Collecting opencv-python
  Using cached opencv_python-4.5.2.54-cp38-cp38-manylinux2014_x86_64.whl (51.0 MB)
Collecting flake8
  Using cached flake8-3.9.2-py2.py3-none-any.whl (73 kB)
Collecting black
  Using cached black-21.6b0-py3-none-any.whl (140 kB)
Collecting pre-commit
  Using cached pre_commit-2.13.0-py2.py3-none-any.whl (190 kB)
Collecting numpy
  Using cached numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.7-py3-none-any.whl (63 kB)
Collecting smmap<5,>=3.0.1
  Using cached smmap-4.0.0-py2.py3-none-any.whl (24 kB)
Collecting cloudpickle<1.7.0,>=1.2.0
  Using cached cloudpickle-1.6.0-py3-none-any.whl (23 kB)
Collecting Pillow<=8.2.0
  Using cached Pillow-8.2.0-cp38-cp38-manylinux1_x86_64.whl (3.0 MB)
Collecting pyglet<=1.5.15,>=1.4.0
  Using cached pyglet-1.5.15-py3-none-any.whl (1.1 MB)
Collecting scipy
  Using cached scipy-1.6.3-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
Collecting pycodestyle<2.8.0,>=2.7.0
  Using cached pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB)
Collecting mccabe<0.7.0,>=0.6.0
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.4.0,>=2.3.0
  Using cached pyflakes-2.3.1-py2.py3-none-any.whl (68 kB)
Collecting pathspec<1,>=0.8.1
  Using cached pathspec-0.8.1-py2.py3-none-any.whl (28 kB)
Collecting toml>=0.10.1
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting click>=7.1.2
  Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting appdirs
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting regex>=2020.1.8
  Using cached regex-2021.4.4-cp38-cp38-manylinux2014_x86_64.whl (733 kB)
Collecting pyyaml>=5.1
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting nodeenv>=0.11.1
  Using cached nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
Collecting cfgv>=2.0.0
  Using cached cfgv-3.3.0-py2.py3-none-any.whl (7.3 kB)
Collecting virtualenv>=20.0.8
  Using cached virtualenv-20.4.7-py2.py3-none-any.whl (7.2 MB)
Collecting identify>=1.0.0
  Using cached identify-2.2.10-py2.py3-none-any.whl (98 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.2-py2.py3-none-any.whl (338 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Installing collected packages: numpy, smmap, six, scipy, pyglet, Pillow, filelock, distlib, cloudpickle, appdirs, virtualenv, toml, regex, pyyaml, pyflakes, pycodestyle, pathspec, opencv-python, nodeenv, mypy-extensions, mccabe, identify, gym, gitdb, click, cfgv, atari-py, pre-commit, gitpython, flake8, black
Successfully installed Pillow-8.2.0 appdirs-1.4.4 atari-py-0.2.5 black-21.6b0 cfgv-3.3.0 click-8.0.1 cloudpickle-1.6.0 distlib-0.3.2 filelock-3.0.12 flake8-3.9.2 gitdb-4.0.7 gitpython-3.1.18 gym-0.18.3 identify-2.2.10 mccabe-0.6.1 mypy-extensions-0.4.3 nodeenv-1.6.0 numpy-1.20.3 opencv-python-4.5.2.54 pathspec-0.8.1 pre-commit-2.13.0 pycodestyle-2.7.0 pyflakes-2.3.1 pyglet-1.5.15 pyyaml-5.4.1 regex-2021.4.4 scipy-1.6.3 six-1.16.0 smmap-4.0.0 toml-0.10.2 virtualenv-20.4.7

adyomin@DLW ~/s/torchbeast (master)> pip install nest/                                                                                                              (nle_38) 
Processing ./nest
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Collecting pybind11>=2.3
  Using cached pybind11-2.6.2-py2.py3-none-any.whl (191 kB)
Building wheels for collected packages: nest
  Building wheel for nest (setup.py) ... done
  Created wheel for nest: filename=nest-0.0.3-cp38-cp38-linux_x86_64.whl size=1359981 sha256=51687f8822780c0a48902e3929ecb001da9095838828c9142f849901d17343d1
  Stored in directory: /tmp/pip-ephem-wheel-cache-isfh3ukc/wheels/3b/d0/ff/525487a4ac7f26e39949ecdbe566c3ac2a9fbc6b7eabd702a9
Successfully built nest
Installing collected packages: pybind11, nest
Successfully installed nest-0.0.3 pybind11-2.6.2

adyomin@DLW ~/s/torchbeast (master)> python setup.py install                                                                                                        (nle_38) 
running install
running bdist_egg
running egg_info
writing libtorchbeast.egg-info/PKG-INFO
writing dependency_links to libtorchbeast.egg-info/dependency_links.txt
writing requirements to libtorchbeast.egg-info/requires.txt
writing top-level names to libtorchbeast.egg-info/top_level.txt
reading manifest file 'libtorchbeast.egg-info/SOURCES.txt'
writing manifest file 'libtorchbeast.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- Could NOT find Python3 (missing: Python3_EXECUTABLE Python3_INCLUDE_DIRS Python3_LIBRARIES Python3_NumPy_INCLUDE_DIRS Interpreter Development NumPy Development.Module Development.Embed) (Required is exact version "3.7")
    Reason given by package: 
        Interpreter: Wrong version for the interpreter "/home/adyomin/miniconda3/envs/nle_38/bin/python3"

-- pybind11 v2.6.2 
-- Found PythonInterp: /home/adyomin/miniconda3/envs/nle_38/bin/python (found version "3.8.10") 
-- Found PythonLibs: /home/adyomin/miniconda3/envs/nle_38/lib
-- 
-- 3.14.0.0
-- Caffe2: CUDA detected: 11.3
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.3
-- Found cuDNN: v8.2.1  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 1ea278b5
-- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
CMake Warning at /home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:51 (find_package)

-- Configuring done
CMake Error at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Target "_C" links to target "Python3::NumPy" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

CMake Error at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Target "_C" links to target "Python3::NumPy" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

CMake Warning at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Cannot generate a safe runtime search path for target _C because there is a
  cycle in the constraint graph:

    dir 0 is [/home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/lib]
    dir 1 is [/usr/local/cuda/lib64/stubs]
    dir 2 is [/usr/local/cuda/lib64]
      dir 3 must precede it due to runtime library [libnvToolsExt.so.1]
    dir 3 is [/home/adyomin/miniconda3/envs/nle/lib]
      dir 2 must precede it due to runtime library [libcudart.so.11.0]
    dir 4 is [/home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages/torch/lib]

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

Output of the python collect_env.py in the same conda environment:

Collecting environment information...
NLE version: N/A
PyTorch version: 1.9.0
Is debug build: No
CUDA used to build PyTorch: 11.1

OS: Ubuntu 21.04
GCC version: (Ubuntu 9.3.0-23ubuntu2) 9.3.0
CMake version: version 3.18.4

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070 SUPER
Nvidia driver version: 465.19.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.1

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.9.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
heiner commented 3 years ago

Hey Andrei,

Thanks for your interest in TorchBeast and for all bug report.

It took me a while to interpret the cmake output here.

I think what happened is that cmake at some point had issues selecting the right Python version out of several, so I fixed it to 3.7. See this line in CMakeLists.txt:

find_package(Python3 3.7 EXACT COMPONENTS Interpreter Development NumPy)

I believe if you were to remove the word EXACT here it might work?

adyomin commented 3 years ago

The suggested edit does indeed allow the build & installation to complete (with some warnings):

adyomin@DLW ~/s/torchbeast (master)> python setup.py install                                                                                                        (nle_38) 
running install
running bdist_egg
running egg_info
writing libtorchbeast.egg-info/PKG-INFO
writing dependency_links to libtorchbeast.egg-info/dependency_links.txt
writing requirements to libtorchbeast.egg-info/requires.txt
writing top-level names to libtorchbeast.egg-info/top_level.txt
reading manifest file 'libtorchbeast.egg-info/SOURCES.txt'
writing manifest file 'libtorchbeast.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- Found Python3: /home/adyomin/miniconda3/envs/nle_38/bin/python3.8 (found suitable version "3.8.10", minimum required is "3.7") found components: Interpreter Development NumPy Development.Module Development.Embed 
-- pybind11 v2.6.2 
-- 
-- 3.14.0.0
-- Caffe2: CUDA detected: 11.3
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.3
-- Found cuDNN: v8.2.1  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 1ea278b5
-- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
CMake Warning at /home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:51 (find_package)

-- Configuring done
CMake Warning at /usr/share/cmake-3.18/Modules/FindPython/Support.cmake:3128 (add_library):
  Cannot generate a safe runtime search path for target _C because there is a
  cycle in the constraint graph:

    dir 0 is [/home/adyomin/miniconda3/envs/nle/lib/python3.8/site-packages/torch/lib]
    dir 1 is [/usr/local/cuda/lib64/stubs]
    dir 2 is [/usr/local/cuda/lib64]
      dir 3 must precede it due to runtime library [libnvToolsExt.so.1]
    dir 3 is [/home/adyomin/miniconda3/envs/nle/lib]
      dir 2 must precede it due to runtime library [libcudart.so.11.0]
    dir 4 is [/home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages/torch/lib]

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  /usr/share/cmake-3.18/Modules/FindPython3.cmake:393 (__Python3_add_library)
  third_party/pybind11/tools/pybind11NewTools.cmake:196 (python3_add_library)
  CMakeLists.txt:96 (pybind11_add_module)

-- Generating done
-- Build files have been written to: /home/adyomin/source/torchbeast/build/temp.linux-x86_64-3.8
[847/1318] Building CXX object grpc/third_party/protobuf/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/message_lite.cc.o
In file included from /usr/include/string.h:519,
                 from ../../third_party/grpc/third_party/protobuf/src/google/protobuf/stubs/port.h:39,
                 from ../../third_party/grpc/third_party/protobuf/src/google/protobuf/stubs/macros.h:34,
                 from ../../third_party/grpc/third_party/protobuf/src/google/protobuf/stubs/common.h:46,
                 from ../../third_party/grpc/third_party/protobuf/src/google/protobuf/message_lite.h:45,
                 from ../../third_party/grpc/third_party/protobuf/src/google/protobuf/message_lite.cc:36:
In function ‘void* memcpy(void*, const void*, size_t)’,
    inlined from ‘google::protobuf::uint8* google::protobuf::io::EpsCopyOutputStream::WriteRaw(const void*, int, google::protobuf::uint8*)’ at ../../third_party/grpc/third_party/protobuf/src/google/protobuf/io/coded_stream.h:699:16,
    inlined from ‘bool google::protobuf::MessageLite::SerializePartialToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const’ at ../../third_party/grpc/third_party/protobuf/src/google/protobuf/implicit_weak_message.h:85:28:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: warning: ‘void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)’ specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   29 |   return __builtin___memcpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   30 |      __glibc_objsize0 (__dest));
      |      ~~~~~~~~~~~~~~~~~~~~~~~~~~  
[1318/1318] Linking CXX shared module ../lib.linux-x86_64-3.8/libtorchbeast/_C.cpython-38-x86_64-linux-gnu.so
[0/1] Install the project...
-- Install configuration: "Release"
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/libtorchbeast
copying build/lib.linux-x86_64-3.8/libtorchbeast/_C.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/libtorchbeast
copying build/lib.linux-x86_64-3.8/libtorchbeast/__init__.py -> build/bdist.linux-x86_64/egg/libtorchbeast
byte-compiling build/bdist.linux-x86_64/egg/libtorchbeast/__init__.py to __init__.cpython-38.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying libtorchbeast.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying libtorchbeast.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying libtorchbeast.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying libtorchbeast.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying libtorchbeast.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying libtorchbeast.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
creating 'dist/libtorchbeast-0.0.20-py3.8-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing libtorchbeast-0.0.20-py3.8-linux-x86_64.egg
creating /home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages/libtorchbeast-0.0.20-py3.8-linux-x86_64.egg
Extracting libtorchbeast-0.0.20-py3.8-linux-x86_64.egg to /home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages
Adding libtorchbeast 0.0.20 to easy-install.pth file

Installed /home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages/libtorchbeast-0.0.20-py3.8-linux-x86_64.egg
Processing dependencies for libtorchbeast==0.0.20
Searching for torch==1.9.0
Best match: torch 1.9.0
Adding torch 1.9.0 to easy-install.pth file
Installing convert-caffe2-to-onnx script to /home/adyomin/miniconda3/envs/nle_38/bin
Installing convert-onnx-to-caffe2 script to /home/adyomin/miniconda3/envs/nle_38/bin

Using /home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages
Searching for typing-extensions==3.7.4.3
Best match: typing-extensions 3.7.4.3
Adding typing-extensions 3.7.4.3 to easy-install.pth file

Using /home/adyomin/miniconda3/envs/nle_38/lib/python3.8/site-packages
Finished processing dependencies for libtorchbeast==0.0.20
heiner commented 3 years ago

Great!

How about we change your PR to just remove EXACT then and see if that causes further problems down the line (which I think it will, for people with several Python versions installed ...)

heiner commented 3 years ago

Thanks, merged.

maxerbubba commented 3 years ago

I was getting similar error on MacOS OSX 11.4 big sur, with conda. Upgrading to python3.8 fixed it.

(nethack) 88665a14b754:torchbeast maxreede$ python setup.py install
running install
running bdist_egg
running egg_info
writing libtorchbeast.egg-info/PKG-INFO
writing dependency_links to libtorchbeast.egg-info/dependency_links.txt
writing requirements to libtorchbeast.egg-info/requires.txt
writing top-level names to libtorchbeast.egg-info/top_level.txt
reading manifest file 'libtorchbeast.egg-info/SOURCES.txt'
writing manifest file 'libtorchbeast.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_py
running build_ext
-- Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found suitable version "3.9.4", minimum required is "3.7")
-- pybind11 v2.6.2 
-- 
-- 3.14.0.0
CMake Deprecation Warning at third_party/grpc/third_party/zlib/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/grpc/third_party/googletest/CMakeLists.txt:4 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/grpc/third_party/googletest/googlemock/CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/grpc/third_party/googletest/googletest/CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Configuring done
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/Volumes/workplace/ml/aicrowd/torchbeast/_Python3_NumPy_INCLUDE_DIR
   used as include directory in directory /Volumes/workplace/ml/aicrowd/torchbeast

CMake Error at third_party/pybind11/tools/pybind11Tools.cmake:166 (add_library):
  Target "_C" links to target "Python3::NumPy" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:96 (pybind11_add_module)

CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   zlib

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
heiner commented 3 years ago

Hey Max,

Thanks for your comment.

It's always a bit hard to parse these CMake errors, but I believe what was missing in your Python 3.9 installation was Numpy. Perhaps there is some issue where CMake has a harder time finding Numpy for Python 3.9, or perhaps it was just not installed in your case?

maxerbubba commented 3 years ago

Don't worry it is working now :) But the trick was that my local conda python was 3.6, so cmake somehow found my global system python (homebrew upgraded it to 3.9) but my system python does not have numpy. So a bit of a red herring.

Upgrading my local conda python to python=3.8 fixed the issue.

On Mon, Jul 5, 2021 at 1:41 AM Heinrich Kuttler @.***> wrote:

Hey Max,

Thanks for your comment.

It's always a bit hard to parse these CMake errors, but I believe what was missing in your Python 3.9 installation was Numpy. Perhaps there is some issue where CMake has a harder time finding Numpy for Python 3.9, or perhaps it was just not installed in your case?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/torchbeast/issues/30#issuecomment-873923667, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7KYTT3CKYKWBRPWQJTS4DTWFVZ7ANCNFSM465M7QHQ .

zh794390558 commented 1 year ago

apt install python3.8-dev