pip install runtime error: Couldn't compile and install ats.ops.extract_patches.libpatches

Jaiczay commented 4 years ago

On Ubuntu 19.04 Python 3.7.3

pip3 install attention-sampling
Collecting attention-sampling
  Using cached https://files.pythonhosted.org/packages/ff/2d/4474d1f516865eb83419c67045d885adc03266f97858fa8eeb14117c709d/attention-sampling-0.2.tar.gz
Requirement already satisfied: keras>=2 in /home/jaiczay/.local/lib/python3.7/site-packages (from attention-sampling) (2.2.4)
Requirement already satisfied: numpy in /home/jaiczay/.local/lib/python3.7/site-packages (from attention-sampling) (1.16.4)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from keras>=2->attention-sampling) (3.13)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from keras>=2->attention-sampling) (1.12.0)
Requirement already satisfied: h5py in /home/jaiczay/.local/lib/python3.7/site-packages (from keras>=2->attention-sampling) (2.9.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /home/jaiczay/.local/lib/python3.7/site-packages (from keras>=2->attention-sampling) (1.1.0)
Requirement already satisfied: scipy>=0.14 in /home/jaiczay/.local/lib/python3.7/site-packages (from keras>=2->attention-sampling) (1.3.1)
Requirement already satisfied: keras-applications>=1.0.6 in /home/jaiczay/.local/lib/python3.7/site-packages (from keras>=2->attention-sampling) (1.0.7)
Building wheels for collected packages: attention-sampling
  Running setup.py bdist_wheel for attention-sampling ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-pbfqhw21/attention-sampling/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-axu0ud85 --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/ats
  copying ats/__init__.py -> build/lib.linux-x86_64-3.7/ats
  creating build/lib.linux-x86_64-3.7/ats/ops
  copying ats/ops/__init__.py -> build/lib.linux-x86_64-3.7/ats/ops
  creating build/lib.linux-x86_64-3.7/ats/utils
  copying ats/utils/layers.py -> build/lib.linux-x86_64-3.7/ats/utils
  copying ats/utils/regularizers.py -> build/lib.linux-x86_64-3.7/ats/utils
  copying ats/utils/training.py -> build/lib.linux-x86_64-3.7/ats/utils
  copying ats/utils/__init__.py -> build/lib.linux-x86_64-3.7/ats/utils
  creating build/lib.linux-x86_64-3.7/ats/core
  copying ats/core/sampling.py -> build/lib.linux-x86_64-3.7/ats/core
  copying ats/core/builder.py -> build/lib.linux-x86_64-3.7/ats/core
  copying ats/core/expectation.py -> build/lib.linux-x86_64-3.7/ats/core
  copying ats/core/ats_layer.py -> build/lib.linux-x86_64-3.7/ats/core
  copying ats/core/__init__.py -> build/lib.linux-x86_64-3.7/ats/core
  creating build/lib.linux-x86_64-3.7/ats/data
  copying ats/data/from_tensors.py -> build/lib.linux-x86_64-3.7/ats/data
  copying ats/data/base.py -> build/lib.linux-x86_64-3.7/ats/data
  copying ats/data/__init__.py -> build/lib.linux-x86_64-3.7/ats/data
  creating build/lib.linux-x86_64-3.7/ats/ops/extract_patches
  copying ats/ops/extract_patches/__init__.py -> build/lib.linux-x86_64-3.7/ats/ops/extract_patches
  running build_ext
  Building ats.ops.extract_patches.libpatches
  -- The C compiler identification is GNU 8.3.0
  -- The CXX compiler identification is GNU 8.3.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  CUDA_TOOLKIT_ROOT_DIR not found or specified
  -- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)
    File "<string>", line 1
      import tensorflow as tf;         print(tf.sysconfig.get_include(), end='')
                                                                            ^
  SyntaxError: invalid syntax
  -- Found TensorFlow include:
    File "<string>", line 1
      import tensorflow as tf;         print(tf.sysconfig.get_lib(), end='')
                                                                        ^
  SyntaxError: invalid syntax
    File "<string>", line 1
      import tensorflow as tf;         lib = next(f for f in tf.sysconfig.get_link_flags()                    if f.startswith('-l'));         print(lib[2:] if lib[2] != ':' else lib[3:], end='')
                                                                                                                                                                                              ^
  SyntaxError: invalid syntax
  -- Found TensorFlow lib: TENSORFLOW_LIB-NOTFOUND
    File "<string>", line 1
      import tensorflow as tf;                 print(tf.__version__, end='')
                                                                        ^
  SyntaxError: invalid syntax
  CMake Warning at /tmp/pip-install-pbfqhw21/attention-sampling/ats/ops/cmake/FindTensorFlow.cmake:60 (message):
    You are using an unsupported compiler! It is recommended to build your
    TensorFlow extensions with gcc 4.8 .
  Call Stack (most recent call first):
    CMakeLists.txt:14 (find_package)

    File "<string>", line 1
      import tensorflow as tf;         print(' '.join(             f for f in tf.sysconfig.get_compile_flags()             if not f.startswith('-I')), end='')
                                                                                                                                                          ^
  SyntaxError: invalid syntax
  -- Added TensorFlow flags:
  CMake Error at /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
    Could NOT find TensorFlow (missing: TENSORFLOW_INCLUDE_DIR TENSORFLOW_LIB)
  Call Stack (most recent call first):
    /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
    /tmp/pip-install-pbfqhw21/attention-sampling/ats/ops/cmake/FindTensorFlow.cmake:82 (find_package_handle_standard_args)
    CMakeLists.txt:14 (find_package)

  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-install-pbfqhw21/attention-sampling/ats/ops/extract_patches/build/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 33, in compile
      cwd=extension_dir
    File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '-DCMAKE_BUILD_TYPE=Release', '..']' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 144, in <module>
      setup_package()
    File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 139, in setup_package
      cmdclass={"build_ext": custom_build_ext}
    File "/home/jaiczay/.local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 188, in run
      self.run_command('build')
    File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/jaiczay/.local/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 78, in run
      _build_ext.run(self)
    File "/usr/lib/python3.7/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/usr/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
      self._build_extensions_serial()
    File "/usr/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 52, in build_extension
      ext.compile()
    File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 46, in compile
      ) from e
  RuntimeError: Couldn't compile and install ats.ops.extract_patches.libpatches

  ----------------------------------------
  Failed building wheel for attention-sampling
  Running setup.py clean for attention-sampling
Failed to build attention-sampling
Installing collected packages: attention-sampling
  Running setup.py install for attention-sampling ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-pbfqhw21/attention-sampling/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-n262zdr0/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/ats
    copying ats/__init__.py -> build/lib.linux-x86_64-3.7/ats
    creating build/lib.linux-x86_64-3.7/ats/ops
    copying ats/ops/__init__.py -> build/lib.linux-x86_64-3.7/ats/ops
    creating build/lib.linux-x86_64-3.7/ats/utils
    copying ats/utils/layers.py -> build/lib.linux-x86_64-3.7/ats/utils
    copying ats/utils/regularizers.py -> build/lib.linux-x86_64-3.7/ats/utils
    copying ats/utils/training.py -> build/lib.linux-x86_64-3.7/ats/utils
    copying ats/utils/__init__.py -> build/lib.linux-x86_64-3.7/ats/utils
    creating build/lib.linux-x86_64-3.7/ats/core
    copying ats/core/sampling.py -> build/lib.linux-x86_64-3.7/ats/core
    copying ats/core/builder.py -> build/lib.linux-x86_64-3.7/ats/core
    copying ats/core/expectation.py -> build/lib.linux-x86_64-3.7/ats/core
    copying ats/core/ats_layer.py -> build/lib.linux-x86_64-3.7/ats/core
    copying ats/core/__init__.py -> build/lib.linux-x86_64-3.7/ats/core
    creating build/lib.linux-x86_64-3.7/ats/data
    copying ats/data/from_tensors.py -> build/lib.linux-x86_64-3.7/ats/data
    copying ats/data/base.py -> build/lib.linux-x86_64-3.7/ats/data
    copying ats/data/__init__.py -> build/lib.linux-x86_64-3.7/ats/data
    creating build/lib.linux-x86_64-3.7/ats/ops/extract_patches
    copying ats/ops/extract_patches/__init__.py -> build/lib.linux-x86_64-3.7/ats/ops/extract_patches
    running build_ext
    Building ats.ops.extract_patches.libpatches
    CUDA_TOOLKIT_ROOT_DIR not found or specified
    -- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)
      File "<string>", line 1
        import tensorflow as tf;         print(tf.sysconfig.get_include(), end='')
                                                                              ^
    SyntaxError: invalid syntax
    -- Found TensorFlow include:
      File "<string>", line 1
        import tensorflow as tf;         print(tf.sysconfig.get_lib(), end='')
                                                                          ^
    SyntaxError: invalid syntax
      File "<string>", line 1
        import tensorflow as tf;         lib = next(f for f in tf.sysconfig.get_link_flags()                    if f.startswith('-l'));         print(lib[2:] if lib[2] != ':' else lib[3:], end='')
                                                                                                                                                                                                ^
    SyntaxError: invalid syntax
    -- Found TensorFlow lib: TENSORFLOW_LIB-NOTFOUND
      File "<string>", line 1
        import tensorflow as tf;                 print(tf.__version__, end='')
                                                                          ^
    SyntaxError: invalid syntax
    CMake Warning at /tmp/pip-install-pbfqhw21/attention-sampling/ats/ops/cmake/FindTensorFlow.cmake:60 (message):
      You are using an unsupported compiler! It is recommended to build your
      TensorFlow extensions with gcc 4.8 .
    Call Stack (most recent call first):
      CMakeLists.txt:14 (find_package)

      File "<string>", line 1
        import tensorflow as tf;         print(' '.join(             f for f in tf.sysconfig.get_compile_flags()             if not f.startswith('-I')), end='')
                                                                                                                                                            ^
    SyntaxError: invalid syntax
    -- Added TensorFlow flags:
    CMake Error at /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
      Could NOT find TensorFlow (missing: TENSORFLOW_INCLUDE_DIR TENSORFLOW_LIB)
    Call Stack (most recent call first):
      /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
      /tmp/pip-install-pbfqhw21/attention-sampling/ats/ops/cmake/FindTensorFlow.cmake:82 (find_package_handle_standard_args)
      CMakeLists.txt:14 (find_package)

    -- Configuring incomplete, errors occurred!
    See also "/tmp/pip-install-pbfqhw21/attention-sampling/ats/ops/extract_patches/build/CMakeFiles/CMakeOutput.log".
    Traceback (most recent call last):
      File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 33, in compile
        cwd=extension_dir
      File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '-DCMAKE_BUILD_TYPE=Release', '..']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 144, in <module>
        setup_package()
      File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 139, in setup_package
        cmdclass={"build_ext": custom_build_ext}
      File "/home/jaiczay/.local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/jaiczay/.local/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.7/distutils/command/install.py", line 589, in run
        self.run_command('build')
      File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/jaiczay/.local/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 78, in run
        _build_ext.run(self)
      File "/usr/lib/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/usr/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/usr/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 52, in build_extension
        ext.compile()
      File "/tmp/pip-install-pbfqhw21/attention-sampling/setup.py", line 46, in compile
        ) from e
    RuntimeError: Couldn't compile and install ats.ops.extract_patches.libpatches

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-pbfqhw21/attention-sampling/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-n262zdr0/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-pbfqhw21/attention-sampling/

angeloskath commented 4 years ago

Hi,

Thanks a lot for the issue! Could you run the following to double-check what the error is?

python -c "import tensorflow as tf; print(tf.sysconfig.get_include(), end='')"

I will try to replicate the issue and have it fixed.

Cheers, Angelos

Jaiczay commented 4 years ago

Thx for the reply. This is the output:

/home/jaiczay/.local/lib/python3.7/site-packages/tensorflow/include

angeloskath commented 4 years ago

Hi again,

I am struggling to reproduce the error, however because several people are having trouble compiling the tf extension. I will try to create precompiled binary distributions for both anaconda and PyPI by the end of the month.

For now, I would suggest

using tensorflow 1.13.1
making sure that you have
- CMake installed
- g++ installed
- tensorflow installed

Let me know if you need anything more.

Cheers, Angelos

lone17 commented 4 years ago

I had the same problem and it f*ck me up for 2 days but i finally got a successful installation. The cause of this problem may vary and it looks like mine is different from yours, but here are some suggestions:

i'm using tf 1.13.1 and python 3.6.8 and cuda 10.0. based on my experience, i recommend you stick with python 3.6 instead of 3.7 when working with tf and cuda
make sure you can use tensorflow, i.e. be able to import tensorflow
if you are using tensorflow-gpu, make sure the cuda versions form nvcc --version and nvidia-smi are the same
try cloning the repo and build it manually, i.e. run python setup.py install. you might also want to add this line set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") to the start of attention-sampling/ats/ops/extract_patches/CMakeLists.txt, i add it at line 6.

best of luck !

idiap / attention-sampling

pip install runtime error: Couldn't compile and install ats.ops.extract_patches.libpatches #3