search

csukuangfj / optimized_transducer

Memory efficient transducer loss computation
Other
68 stars 11 forks source link

Issue with optimized-transducer installation #25

Closed AmirHussein96 closed 2 years ago

AmirHussein96 commented 2 years ago

I started installing K2, lhotse and Icefall. So far I was able to test K2 and it works perfectly, lhotse also works but when I tried to install icefall I got a weird issue about optimized-transducer. The log is below.

Collecting kaldilm Using cached kaldilm-1.11-cp38-cp38-linux_x86_64.whl Collecting kaldialign Using cached kaldialign-0.2-cp38-cp38-linux_x86_64.whl Requirement already satisfied: sentencepiece>=0.1.96 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (0.1.96) Requirement already satisfied: tensorboard in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (2.7.0) Requirement already satisfied: typeguard in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (2.13.3) Collecting optimized_transducer Using cached optimized_transducer-1.3.tar.gz (47 kB) Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (1.8.1) Requirement already satisfied: werkzeug>=0.11.15 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (2.0.2) Requirement already satisfied: numpy>=1.12.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (1.21.2) Requirement already satisfied: protobuf>=3.6.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (3.19.3) Requirement already satisfied: wheel>=0.26 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (0.37.1) Requirement already satisfied: setuptools>=41.0.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (58.0.4) Requirement already satisfied: grpcio>=1.24.3 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (1.43.0) Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (0.4.6) Requirement already satisfied: absl-py>=0.4 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (1.0.0) Requirement already satisfied: google-auth<3,>=1.6.3 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (2.3.3) Requirement already satisfied: requests<3,>=2.21.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (2.27.1) Requirement already satisfied: markdown>=2.6.8 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (3.3.6) Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from tensorboard->-r requirements.txt (line 4)) (0.6.1) Requirement already satisfied: six in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from absl-py>=0.4->tensorboard->-r requirements.txt (line 4)) (1.16.0) Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 4)) (4.2.4) Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 4)) (0.2.8) Requirement already satisfied: rsa<5,>=3.1.4 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 4)) (4.8) Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard->-r requirements.txt (line 4)) (1.3.0) Requirement already satisfied: importlib-metadata>=4.4 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard->-r requirements.txt (line 4)) (4.10.1) Requirement already satisfied: zipp>=0.5 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard->-r requirements.txt (line 4)) (3.7.0) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 4)) (0.4.8) Requirement already satisfied: charset-normalizer~=2.0.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard->-r requirements.txt (line 4)) (2.0.10) Requirement already satisfied: certifi>=2017.4.17 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard->-r requirements.txt (line 4)) (2021.10.8) Requirement already satisfied: idna<4,>=2.5 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard->-r requirements.txt (line 4)) (3.3) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard->-r requirements.txt (line 4)) (1.26.8) Requirement already satisfied: oauthlib>=3.0.0 in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard->-r requirements.txt (line 4)) (3.1.1) Building wheels for collected packages: optimized-transducer Building wheel for optimized-transducer (setup.py): started Building wheel for optimized-transducer (setup.py): finished with status 'error' ERROR: Command errored out with exit status 1: command: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py'"'"'; file='"'"'/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-qa004082 cwd: /tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/ Complete output (153 lines): running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/init.py -> build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/transducer_loss.py -> build/lib.linux-x86_64-3.8/optimized_transducer running build_ext For fast compilation, run: export OT_MAKE_ARGS="-j"; python setup.py install Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python build command is:

          cd build/temp.linux-x86_64-3.8

          cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python /tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173

          make  _optimized_transducer

-- Enabled languages: CXX;CUDA -- The CXX compiler identification is GNU 6.5.0 -- The CUDA compiler identification is NVIDIA 11.1.74 -- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++ -- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Check for working CUDA compiler: /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc -- Check for working CUDA compiler: /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc -- works -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- Automatic GPU detection failed. Building for common architectures. -- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;8.0;8.6;8.6+PTX -- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86 -- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75;80;86 -- Adding arch 35 -- Adding arch 50 -- Adding arch 60 -- Adding arch 61 -- Adding arch 70 -- Adding arch 75 -- Adding arch 80 -- Adding arch 86 -- OT_COMPUTE_ARCHS: 35;50;60;61;70;75;80;86 -- Downloading pybind11 -- pybind11 is downloaded to /tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/build/temp.linux-x86_64-3.8/_deps/pybind11-src -- pybind11 v2.6.0 -- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12") -- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so -- Performing Test HAS_FLTO -- Performing Test HAS_FLTO - Success -- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -- Looking for C++ include pthread.h -- Looking for C++ include pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package): Policy CMP0074 is not set: find_package uses _ROOT variables. Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

  /cm/shared/apps/cuda11.1/toolkit/11.1.0

For compatibility, CMake is ignoring the variable.

Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:130 (include) This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda11.1/toolkit/11.1.0 (found version "11.1") -- Caffe2: CUDA detected: 11.1 -- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc -- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda11.1/toolkit/11.1.0 -- Caffe2: Header version is: 11.1 -- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) CMake Warning at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:111 (message): Caffe2: Cannot find cuDNN library. Turning the option off Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:130 (include)

-- /cm/shared/apps/cuda11.1/toolkit/11.1.0/lib64/libnvrtc.so shorthash is 1f6b333a -- Automatic GPU detection failed. Building for common architectures. -- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;8.0;8.6;8.6+PTX -- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86 CMake Error at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message): Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN libraries. Please set the proper cuDNN prefixes and / or install cuDNN. Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:130 (include)

-- Configuring incomplete, errors occurred! See also "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". See also "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeError.log". make: * No rule to make target `_optimized_transducer'. Stop. Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py", line 101, in setuptools.setup( File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(attrs) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run self.run_command('build') File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py", line 60, in build_extension raise Exception( Exception: Build optimized_transducer failed. Please check the error message. You can ask for help by creating an issue on GitHub.

Click: https://github.com/csukuangfj/optimized_transducer/issues/new

ERROR: Failed building wheel for optimized-transducer Running setup.py clean for optimized-transducer Failed to build optimized-transducer Installing collected packages: optimized-transducer, kaldilm, kaldialign Running setup.py install for optimized-transducer: started Running setup.py install for optimized-transducer: finished with status 'error' ERROR: Command errored out with exit status 1: command: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py'"'"'; file='"'"'/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-mcbah0p8/install-record.txt --single-version-externally-managed --compile --install-headers /home/local/QCRI/ahussein/anaconda3/envs/k2/include/python3.8/optimized-transducer cwd: /tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/ Complete output (155 lines): running install running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/init.py -> build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/transducer_loss.py -> build/lib.linux-x86_64-3.8/optimized_transducer running build_ext For fast compilation, run: export OT_MAKE_ARGS="-j"; python setup.py install Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python build command is:

            cd build/temp.linux-x86_64-3.8

            cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python /tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173

            make  _optimized_transducer

-- Enabled languages: CXX;CUDA
-- The CXX compiler identification is GNU 6.5.0
-- The CUDA compiler identification is NVIDIA 11.1.74
-- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++
-- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working CUDA compiler: /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc
-- Check for working CUDA compiler: /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;8.0;8.6;8.6+PTX
-- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86
-- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75;80;86
-- Adding arch 35
-- Adding arch 50
-- Adding arch 60
-- Adding arch 61
-- Adding arch 70
-- Adding arch 75
-- Adding arch 80
-- Adding arch 86
-- OT_COMPUTE_ARCHS: 35;50;60;61;70;75;80;86
-- Downloading pybind11
-- pybind11 is downloaded to /tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/build/temp.linux-x86_64-3.8/_deps/pybind11-src
-- pybind11 v2.6.0
-- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12")
-- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package):
  Policy CMP0074 is not set: find_package uses <PackageName>_ROOT variables.
  Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  Environment variable CUDA_ROOT is set to:

    /cm/shared/apps/cuda11.1/toolkit/11.1.0

  For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  cmake/torch.cmake:11 (find_package)
  CMakeLists.txt:130 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda11.1/toolkit/11.1.0 (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda11.1/toolkit/11.1.0/bin/nvcc
-- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda11.1/toolkit/11.1.0
-- Caffe2: Header version is: 11.1
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)
CMake Warning at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:111 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  cmake/torch.cmake:11 (find_package)
  CMakeLists.txt:130 (include)

-- /cm/shared/apps/cuda11.1/toolkit/11.1.0/lib64/libnvrtc.so shorthash is 1f6b333a
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;8.0;8.6;8.6+PTX
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86
CMake Error at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
  Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
  libraries.  Please set the proper cuDNN prefixes and / or install cuDNN.
Call Stack (most recent call first):
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  cmake/torch.cmake:11 (find_package)
  CMakeLists.txt:130 (include)

-- Configuring incomplete, errors occurred!
See also "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
See also "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeError.log".
make: *** No rule to make target `_optimized_transducer'.  Stop.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py", line 101, in <module>
    setuptools.setup(
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py", line 60, in build_extension
    raise Exception(
Exception:
Build optimized_transducer failed. Please check the error message.
You can ask for help by creating an issue on GitHub.

Click:
    https://github.com/csukuangfj/optimized_transducer/issues/new

----------------------------------------

ERROR: Command errored out with exit status 1: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py'"'"'; file='"'"'/tmp/pip-install-jw6digfq/optimized-transducer_865f3ecab82f4f25914b71cca4901173/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-mcbah0p8/install-record.txt --single-version-externally-managed --compile --install-headers /home/local/QCRI/ahussein/anaconda3/envs/k2/include/python3.8/optimized-transducer Check the logs for full command output.

csukuangfj commented 2 years ago

Please install cuDNN on your system first.

AmirHussein96 commented 2 years ago

Please install cuDNN on your system first.

Thank you @csukuangfj . Should I install a specific CUDA version? I installed Pytorch 1.8.1 with cudatoolkit 10.2. The GPU device is visible to the library but the installation gives the error below. I installed pytorch with cuda using the following command assuming that it should also install cuDNN internally conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=10.2 -c pytorch. Should I manually install cuDNN?

-- Found CUDA: /usr/local/cuda (found version "10.2") -- Caffe2: CUDA detected: 10.2 -- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc -- Caffe2: CUDA toolkit directory: /usr/local/cuda -- Caffe2: Header version is: 10.2 -- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) CMake Warning at /home/ahussein/miniconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:109 (message): Caffe2: Cannot find cuDNN library. Turning the option off Call Stack (most recent call first): /home/ahussein/miniconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/ahussein/miniconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:130 (include)

csukuangfj commented 2 years ago

I installed pytorch with cuda using the following command assuming that it should also install cuDNN internally conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=10.2 -c pytorch

Could you first locate where libcudnn.so is?

find /home/local/QCRI/ahussein/anaconda3/envs/k2 -name  "libcudnn.so"

If it happens to be in /home/local/QCRI/ahussein/anaconda3/envs/k2/lib, then please use

export OT_CMAKE_ARGS="-DCUDNN_LIBRARY_PATH=/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libcudnn.so -DCUDNN_INCLUDE_PATH=/home/local/QCRI/ahussein/anaconda3/envs/k2/include"
pip install --verbose optimized_transducer
AmirHussein96 commented 2 years ago

Thank you @csukuangfj , I searched for the "libcudnn.so" in the /home/local/QCRI/ahussein/anaconda3/envs/k2 but could not find it, so I searched in the shared apps folder where all the installed cuda libraries are and found it here: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so`. Then I followed your instructions `export OT_CMAKE_ARGS="-DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include" pip install --verbose optimized_transducer.

I got the following:

Using pip 21.2.4 from /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/pip (python 3.8) Collecting optimized_transducer Using cached optimized_transducer-1.3.tar.gz (47 kB) Running command python setup.py egg_info running egg_info creating /tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info writing /tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info/PKG-INFO writing dependency_links to /tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info/dependency_links.txt writing top-level names to /tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info/top_level.txt writing manifest file '/tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info/SOURCES.txt' reading manifest file '/tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file '/tmp/pip-pip-egg-info-ur6950_d/optimized_transducer.egg-info/SOURCES.txt' Building wheels for collected packages: optimized-transducer Building wheel for optimized-transducer (setup.py): started Running command /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"'; file='"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-glthyjby running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/init.py -> build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/transducer_loss.py -> build/lib.linux-x86_64-3.8/optimized_transducer running build_ext For fast compilation, run: export OT_MAKE_ARGS="-j"; python setup.py install Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python build command is:

          cd build/temp.linux-x86_64-3.8

          cmake -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5

          make  _optimized_transducer

-- Enabled languages: CXX;CUDA -- The CXX compiler identification is GNU 6.5.0 -- The CUDA compiler identification is NVIDIA 10.2.89 -- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++ -- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Check for working CUDA compiler: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc -- Check for working CUDA compiler: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc -- works -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- No CMAKE_BUILD_TYPE given, default to Release -- Autodetected CUDA architecture(s): 7.0 -- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70 -- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75 -- Skipping arch 35 -- Skipping arch 50 -- Skipping arch 60 -- Skipping arch 61 -- Adding arch 70 -- Skipping arch 75 -- OT_COMPUTE_ARCHS: 70 -- Downloading pybind11 -- pybind11 is downloaded to /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/build/temp.linux-x86_64-3.8/_deps/pybind11-src -- pybind11 v2.6.0 -- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12") -- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so -- Performing Test HAS_FLTO -- Performing Test HAS_FLTO - Success -- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -- Looking for C++ include pthread.h -- Looking for C++ include pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package): Policy CMP0074 is not set: find_package uses _ROOT variables. Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

  /cm/shared/apps/cuda10.2/toolkit/10.2.89

For compatibility, CMake is ignoring the variable.

Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:130 (include) This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda10.2/toolkit/10.2.89 (found version "10.2") -- Caffe2: CUDA detected: 10.2 -- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc -- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89 -- Caffe2: Header version is: 10.2 -- Found CUDNN: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -- Found cuDNN: v8.0.5 (include: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include, library: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64) -- /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/libnvrtc.so shorthash is 08c4863f -- Autodetected CUDA architecture(s): 7.0 -- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70 -- Found Torch: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib/libtorch.so -- PyTorch version: 1.8.1 -- PyTorch cuda version: 10.2 -- Downloading moderngpu -- moderngpu is downloaded to /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/build/temp.linux-x86_64-3.8/_deps/moderngpu-src -- Configuring done CMake Warning at build/temp.linux-x86_64-3.8/_deps/pybind11-src/tools/pybind11Tools.cmake:147 (add_library): Cannot generate a safe runtime search path for target _optimized_transducer because files in some directories may conflict with libraries in implicit directories:

  runtime library [libcufft.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libcurand.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libcublas.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libnvrtc.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libnvToolsExt.so.1] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libcudart.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib

Some of these libraries may not be found correctly.

Call Stack (most recent call first): optimized_transducer/python/csrc/CMakeLists.txt:3 (pybind11_add_module)

-- Generating done -- Build files have been written to: /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/build/temp.linux-x86_64-3.8 Scanning dependencies of target optimized_transducer_core_cuda [ 11%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cuda.cu.o [ 22%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/kernels.cu.o [ 33%] Linking CUDA device code CMakeFiles/optimized_transducer_core_cuda.dir/cmake_device_link.o nvcc fatal : Unknown option '-Wl,--no-as-needed,/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so' make[3]: [optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cmake_device_link.o] Error 1 make[2]: [optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/all] Error 2 make[1]: [optimized_transducer/python/csrc/CMakeFiles/_optimized_transducer.dir/rule] Error 2 make: [_optimized_transducer] Error 2 Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py", line 101, in setuptools.setup( File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(**attrs) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run self.run_command('build') File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py", line 60, in build_extension raise Exception( Exception: Build optimized_transducer failed. Please check the error message. You can ask for help by creating an issue on GitHub.

Click: https://github.com/csukuangfj/optimized_transducer/issues/new

Building wheel for optimized-transducer (setup.py): finished with status 'error' ERROR: Failed building wheel for optimized-transducer Running setup.py clean for optimized-transducer Running command /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"'; file='"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' clean --all running clean removing 'build/temp.linux-x86_64-3.8' (and everything under it) removing 'build/lib.linux-x86_64-3.8' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.8' does not exist -- can't clean it removing 'build' Failed to build optimized-transducer Installing collected packages: optimized-transducer Running setup.py install for optimized-transducer: started Running command /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"'; file='"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-eqlru09p/install-record.txt --single-version-externally-managed --compile --install-headers /home/local/QCRI/ahussein/anaconda3/envs/k2/include/python3.8/optimized-transducer running install running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/init.py -> build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/transducer_loss.py -> build/lib.linux-x86_64-3.8/optimized_transducer running build_ext For fast compilation, run: export OT_MAKE_ARGS="-j"; python setup.py install Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python build command is:

            cd build/temp.linux-x86_64-3.8

            cmake -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5

            make  _optimized_transducer

-- Enabled languages: CXX;CUDA
-- The CXX compiler identification is GNU 6.5.0
-- The CUDA compiler identification is NVIDIA 10.2.89
-- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++
-- Check for working CXX compiler: /cm/shared/apps/gcc6/6.5.0/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working CUDA compiler: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc
-- Check for working CUDA compiler: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- No CMAKE_BUILD_TYPE given, default to Release
-- Autodetected CUDA architecture(s):  7.0
-- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70
-- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75
-- Skipping arch 35
-- Skipping arch 50
-- Skipping arch 60
-- Skipping arch 61
-- Adding arch 70
-- Skipping arch 75
-- OT_COMPUTE_ARCHS: 70
-- Downloading pybind11
-- pybind11 is downloaded to /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/build/temp.linux-x86_64-3.8/_deps/pybind11-src
-- pybind11 v2.6.0
-- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12")
-- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package):
  Policy CMP0074 is not set: find_package uses <PackageName>_ROOT variables.
  Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  Environment variable CUDA_ROOT is set to:

    /cm/shared/apps/cuda10.2/toolkit/10.2.89

  For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  cmake/torch.cmake:11 (find_package)
  CMakeLists.txt:130 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda10.2/toolkit/10.2.89 (found version "10.2")
-- Caffe2: CUDA detected: 10.2
-- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc
-- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89
-- Caffe2: Header version is: 10.2
-- Found CUDNN: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64
-- Found cuDNN: v8.0.5  (include: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include, library: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64)
-- /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/libnvrtc.so shorthash is 08c4863f
-- Autodetected CUDA architecture(s):  7.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
-- Found Torch: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib/libtorch.so
-- PyTorch version: 1.8.1
-- PyTorch cuda version: 10.2
-- Downloading moderngpu
-- moderngpu is downloaded to /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/build/temp.linux-x86_64-3.8/_deps/moderngpu-src
-- Configuring done
CMake Warning at build/temp.linux-x86_64-3.8/_deps/pybind11-src/tools/pybind11Tools.cmake:147 (add_library):
  Cannot generate a safe runtime search path for target _optimized_transducer
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcurand.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcublas.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcudart.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  optimized_transducer/python/csrc/CMakeLists.txt:3 (pybind11_add_module)

-- Generating done
-- Build files have been written to: /tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/build/temp.linux-x86_64-3.8
Scanning dependencies of target optimized_transducer_core_cuda
[ 11%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cuda.cu.o
[ 22%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/kernels.cu.o
[ 33%] Linking CUDA device code CMakeFiles/optimized_transducer_core_cuda.dir/cmake_device_link.o
nvcc fatal   : Unknown option '-Wl,--no-as-needed,/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so'
make[3]: *** [optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cmake_device_link.o] Error 1
make[2]: *** [optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/all] Error 2
make[1]: *** [optimized_transducer/python/csrc/CMakeFiles/_optimized_transducer.dir/rule] Error 2
make: *** [_optimized_transducer] Error 2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py", line 101, in <module>
    setuptools.setup(
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py", line 60, in build_extension
    raise Exception(
Exception:
Build optimized_transducer failed. Please check the error message.
You can ask for help by creating an issue on GitHub.

Click:
    https://github.com/csukuangfj/optimized_transducer/issues/new

Running setup.py install for optimized-transducer: finished with status 'error'

ERROR: Command errored out with exit status 1: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"'; file='"'"'/tmp/pip-install-uj829du7/optimized-transducer_7d0bb2bc941b4bfa894a4848997015c5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-eqlru09p/install-record.txt --single-version-externally-managed --compile --install-headers /home/local/QCRI/ahussein/anaconda3/envs/k2/include/python3.8/optimized-transducer Check the logs for full command output.

csukuangfj commented 2 years ago

I have this error before with CMake 3.12.

But when I switch to CMake 3.11 or CMake 3.18, the error disappears. Could you try that?

AmirHussein96 commented 2 years ago

I tried CMake 3.18 and it seems like it started the installation, but gave me another issue below. Unfortunately, I do not have Cmake 3.11 ready on the grid, I will try it once it is installed on the system.

Using pip 21.2.4 from /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/pip (python 3.8) Collecting optimized_transducer Using cached optimized_transducer-1.3.tar.gz (47 kB) Running command python setup.py egg_info running egginfo creating /tmp/pip-pip-egg-info-o79k59m/optimizedtransducer.egg-info writing /tmp/pip-pip-egg-info-o79k59m/optimized_transducer.egg-info/PKG-INFO writing dependencylinks to /tmp/pip-pip-egg-info-o79k59m/optimized_transducer.egg-info/dependencylinks.txt writing top-level names to /tmp/pip-pip-egg-info-o79k59m/optimized_transducer.egg-info/toplevel.txt writing manifest file '/tmp/pip-pip-egg-info-o79k59m/optimizedtransducer.egg-info/SOURCES.txt' reading manifest file '/tmp/pip-pip-egg-info-o79k59m/optimizedtransducer.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file '/tmp/pip-pip-egg-info-o79k59m/optimized_transducer.egg-info/SOURCES.txt' Building wheels for collected packages: optimized-transducer Building wheel for optimized-transducer (setup.py): started Running command /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"'; file='"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ux6xbvys running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/init.py -> build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/transducer_loss.py -> build/lib.linux-x86_64-3.8/optimized_transducer running build_ext For fast compilation, run: export OT_MAKE_ARGS="-j"; python setup.py install Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python build command is:

          cd build/temp.linux-x86_64-3.8

          cmake -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1

          make  _optimized_transducer

-- Enabled languages: CXX;CUDA -- The CXX compiler identification is GNU 5.5.0 -- The CUDA compiler identification is NVIDIA 10.2.89 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /cm/shared/apps/gcc5/5.5.0/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- Check for working CUDA compiler: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc - skipped -- Detecting CUDA compile features -- Detecting CUDA compile features - done -- No CMAKE_BUILD_TYPE given, default to Release -- Autodetected CUDA architecture(s): 7.0 -- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70 -- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75 -- Skipping arch 35 -- Skipping arch 50 -- Skipping arch 60 -- Skipping arch 61 -- Adding arch 70 -- Skipping arch 75 -- OT_COMPUTE_ARCHS: 70 -- Downloading pybind11 -- pybind11 is downloaded to /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/build/temp.linux-x86_64-3.8/_deps/pybind11-src -- pybind11 v2.6.0 -- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12") -- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so -- Performing Test HAS_FLTO -- Performing Test HAS_FLTO - Success -- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -- Looking for C++ include pthread.h -- Looking for C++ include pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package): Policy CMP0074 is not set: find_package uses _ROOT variables. Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

  /cm/shared/apps/cuda10.2/toolkit/10.2.89

For compatibility, CMake is ignoring the variable.

Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:130 (include) This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda10.2/toolkit/10.2.89 (found version "10.2") -- Caffe2: CUDA detected: 10.2 -- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc -- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89 -- Caffe2: Header version is: 10.2 -- Found CUDNN: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -- Found cuDNN: v8.0.5 (include: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include, library: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64) -- /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/libnvrtc.so shorthash is 08c4863f -- Autodetected CUDA architecture(s): 7.0 -- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70 -- Found Torch: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib/libtorch.so -- PyTorch version: 1.8.1 -- PyTorch cuda version: 10.2 -- Downloading moderngpu -- moderngpu is downloaded to /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/build/temp.linux-x86_64-3.8/_deps/moderngpu-src -- Configuring done CMake Warning at build/temp.linux-x86_64-3.8/_deps/pybind11-src/tools/pybind11Tools.cmake:147 (add_library): Cannot generate a safe runtime search path for target _optimized_transducer because files in some directories may conflict with libraries in implicit directories:

  runtime library [libcufft.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libcurand.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libcublas.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libnvrtc.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libnvToolsExt.so.1] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
  runtime library [libcudart.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
    /home/local/QCRI/ahussein/anaconda3/envs/k2/lib

Some of these libraries may not be found correctly.

Call Stack (most recent call first): optimized_transducer/python/csrc/CMakeLists.txt:3 (pybind11_add_module)

-- Generating done -- Build files have been written to: /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/build/temp.linux-x86_64-3.8 Scanning dependencies of target optimized_transducer_core_cuda [ 12%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cuda.cu.o [ 25%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/kernels.cu.o [ 37%] Linking CUDA shared library ../../lib/liboptimized_transducer_core_cuda.so /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64: file not recognized: Is a directory collect2: error: ld returned 1 exit status make[3]: [lib/liboptimized_transducer_core_cuda.so] Error 1 make[2]: [optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/all] Error 2 make[1]: [optimized_transducer/python/csrc/CMakeFiles/_optimized_transducer.dir/rule] Error 2 make: [_optimized_transducer] Error 2 Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py", line 101, in setuptools.setup( File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(**attrs) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run self.run_command('build') File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py", line 60, in build_extension raise Exception( Exception: Build optimized_transducer failed. Please check the error message. You can ask for help by creating an issue on GitHub.

Click: https://github.com/csukuangfj/optimized_transducer/issues/new

Building wheel for optimized-transducer (setup.py): finished with status 'error' ERROR: Failed building wheel for optimized-transducer Running setup.py clean for optimized-transducer Running command /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"'; file='"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' clean --all running clean removing 'build/temp.linux-x86_64-3.8' (and everything under it) removing 'build/lib.linux-x86_64-3.8' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.8' does not exist -- can't clean it removing 'build' Failed to build optimized-transducer Installing collected packages: optimized-transducer Running setup.py install for optimized-transducer: started Running command /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"'; file='"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-pd4r86ec/install-record.txt --single-version-externally-managed --compile --install-headers /home/local/QCRI/ahussein/anaconda3/envs/k2/include/python3.8/optimized-transducer running install running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/init.py -> build/lib.linux-x86_64-3.8/optimized_transducer copying optimized_transducer/python/optimized_transducer/transducer_loss.py -> build/lib.linux-x86_64-3.8/optimized_transducer running build_ext For fast compilation, run: export OT_MAKE_ARGS="-j"; python setup.py install Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python build command is:

            cd build/temp.linux-x86_64-3.8

            cmake -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1

            make  _optimized_transducer

-- Enabled languages: CXX;CUDA
-- The CXX compiler identification is GNU 5.5.0
-- The CUDA compiler identification is NVIDIA 10.2.89
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /cm/shared/apps/gcc5/5.5.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- No CMAKE_BUILD_TYPE given, default to Release
-- Autodetected CUDA architecture(s):  7.0
-- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70
-- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75
-- Skipping arch 35
-- Skipping arch 50
-- Skipping arch 60
-- Skipping arch 61
-- Adding arch 70
-- Skipping arch 75
-- OT_COMPUTE_ARCHS: 70
-- Downloading pybind11
-- pybind11 is downloaded to /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/build/temp.linux-x86_64-3.8/_deps/pybind11-src
-- pybind11 v2.6.0
-- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12")
-- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package):
  Policy CMP0074 is not set: find_package uses <PackageName>_ROOT variables.
  Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  Environment variable CUDA_ROOT is set to:

    /cm/shared/apps/cuda10.2/toolkit/10.2.89

  For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  cmake/torch.cmake:11 (find_package)
  CMakeLists.txt:130 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda10.2/toolkit/10.2.89 (found version "10.2")
-- Caffe2: CUDA detected: 10.2
-- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc
-- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89
-- Caffe2: Header version is: 10.2
-- Found CUDNN: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64
-- Found cuDNN: v8.0.5  (include: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include, library: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64)
-- /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/libnvrtc.so shorthash is 08c4863f
-- Autodetected CUDA architecture(s):  7.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
-- Found Torch: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib/libtorch.so
-- PyTorch version: 1.8.1
-- PyTorch cuda version: 10.2
-- Downloading moderngpu
-- moderngpu is downloaded to /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/build/temp.linux-x86_64-3.8/_deps/moderngpu-src
-- Configuring done
CMake Warning at build/temp.linux-x86_64-3.8/_deps/pybind11-src/tools/pybind11Tools.cmake:147 (add_library):
  Cannot generate a safe runtime search path for target _optimized_transducer
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcurand.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcublas.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcudart.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  optimized_transducer/python/csrc/CMakeLists.txt:3 (pybind11_add_module)

-- Generating done
-- Build files have been written to: /tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/build/temp.linux-x86_64-3.8
Scanning dependencies of target optimized_transducer_core_cuda
[ 12%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cuda.cu.o
[ 25%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/kernels.cu.o
[ 37%] Linking CUDA shared library ../../lib/liboptimized_transducer_core_cuda.so
/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64: file not recognized: Is a directory
collect2: error: ld returned 1 exit status
make[3]: *** [lib/liboptimized_transducer_core_cuda.so] Error 1
make[2]: *** [optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/all] Error 2
make[1]: *** [optimized_transducer/python/csrc/CMakeFiles/_optimized_transducer.dir/rule] Error 2
make: *** [_optimized_transducer] Error 2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py", line 101, in <module>
    setuptools.setup(
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py", line 60, in build_extension
    raise Exception(
Exception:
Build optimized_transducer failed. Please check the error message.
You can ask for help by creating an issue on GitHub.

Click:
    https://github.com/csukuangfj/optimized_transducer/issues/new

Running setup.py install for optimized-transducer: finished with status 'error'

ERROR: Command errored out with exit status 1: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"'; file='"'"'/tmp/pip-install-d_kgchow/optimized-transducer_78a65be0658f48fc892619cd964ac2c1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-pd4r86ec/install-record.txt --single-version-externally-managed --compile --install-headers /home/local/QCRI/ahussein/anaconda3/envs/k2/include/python3.8/optimized-transducer Check the logs for full command output.

csukuangfj commented 2 years ago

No, CMake 3.18 is ok.

You only need to change

-DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64

to

-DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so
AmirHussein96 commented 2 years ago

Finally it worked!!! Thank you so so much @csukuangfj for all your help! 

Successfully built optimized-transducer
Installing collected packages: optimized-transducer
Successfully installed optimized-transducer-1.3