search

csukuangfj / kaldifeat

Kaldi-compatible online & offline feature extraction with PyTorch, supporting CUDA, batch processing, chunk processing, and autograd - Provide C++ & Python API
https://csukuangfj.github.io/kaldifeat
Other
186 stars 35 forks source link

Failed to install kaldifeat from source #59

Closed AmirHussein96 closed 1 year ago

AmirHussein96 commented 1 year ago

I tried installing kaldifeat from source since both pip and conda installation failed. I followed the suggestions in FAQ

export KALDIFEAT_CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include"

Then I run the python setup.py install

I could not capture the complete logs but I attached the CMakeOutput.log, and CMakeError.log

The installation started well but then failed for some reason, I managed to capture some of the logs manually because The command python setup.py install &>log gave file with unreadable data.

opying kaldifeat/python/kaldifeat/mfcc.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/fbank.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/offline_feature.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/__init__.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/online_feature.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
running build_ext
Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python
For fast compilation, run:
export KALDIFEAT_MAKE_ARGS="-j"; python setup.py install
Setting make_args to '-j4'
build command is:

                cd build/temp.linux-x86_64-cpython-38

                cmake -DCMAKE_BUILD_TYPE=Release -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -Dkaldifeat_BUILD_TESTS=OFF  -DCMAKE_INSTALL_PREFIX=/alt-arabic/speech/amir/k2/tmp/kaldifeat/build/lib.linux-x86_64-cpython-38/kaldifeat  /alt-arabic/speech/amir/k2/tmp/kaldifeat

                make  -j4  _kaldifeat install

-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /cm/shared/apps/gcc8/8.4.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /cm/shared/apps/gcc8/8.4.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- C++ Standard version: 14
-- Downloading pybind11

  Cannot generate a safe runtime search path for target _kaldifeat because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcurand.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcublas.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcudart.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  kaldifeat/python/csrc/CMakeLists.txt:2 (pybind11_add_module)

-- Generating done
-- Build files have been written to: /alt-arabic/speech/amir/k2/tmp/kaldifeat/build/temp.linux-x86_64-cpython-38
Scanning dependencies of target kaldifeat_core
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-fbank.cc.o
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-functions.cc.o
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-mfcc.cc.o
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-plp.cc.o
[ 25%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-spectrogram.cc.o
[ 30%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-window.cc.o
[ 35%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/matrix-functions.cc.o
[ 40%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/mel-computations.cc.o
[ 45%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/online-feature.cc.o
[ 50%] Linking CXX shared library ../../lib/libkaldifeat_core.so
[ 50%] Built target kaldifeat_core
Scanning dependencies of target _kaldifeat
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-fbank.cc.o
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-mfcc.cc.o
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-plp.cc.o
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-spectrogram.cc.o
[ 80%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/kaldifeat.cc.o
[ 85%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/mel-computations.cc.o
[ 90%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/online-feature.cc.o
[ 95%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/utils.cc.o
[100%] Linking CXX shared module ../../../lib/_kaldifeat.cpython-38-x86_64-linux-gnu.so
[100%] Built target _kaldifeat
[ 45%] Built target kaldifeat_core
Scanning dependencies of target test_kaldifeat
[ 90%] Built target _kaldifeat
[ 95%] Building CXX object kaldifeat/csrc/CMakeFiles/test_kaldifeat.dir/test_kaldifeat.cc.o

 File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run

    _build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
    self._build_extensions_serial()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
    self.build_extension(ext)
  File "/alt-arabic/speech/amir/k2/tmp/kaldifeat/cmake/cmake_extension.py", line 124, in build_extension
    raise Exception(
Exception:
Build kaldifeat failed. Please check the error message.

CMakeOutput.log CMakeError.log

csukuangfj commented 1 year ago

Could you use

git clone https://github.com/csukuangfj/kaldifeat
cd kaldifeat
mkdir build
cd build
cmake ..
make

and post the logs of make?

The above logs you posted are not informative and it is not possible to figure out what bad things happened.

AmirHussein96 commented 1 year ago

Logs of cmake ..

-- The C compiler identification is GNU 4.8.5 -- The CXX compiler identification is GNU 4.8.5 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- No CMAKE_BUILD_TYPE given, default to Release -- C++ Standard version: 14 -- Downloading pybind11 -- pybind11 is downloaded to /alt-arabic/speech/amir/k2/tmp/kaldifeat/build/_deps/pybind11-src -- pybind11 v2.9.2 -- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12") -- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so -- Performing Test HAS_FLTO -- Performing Test HAS_FLTO - Success -- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -- TORCH_DIR: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package): Policy CMP0074 is not set: find_package uses _ROOT variables. Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

/cm/shared/apps/cuda10.2/toolkit/10.2.89

For compatibility, CMake is ignoring the variable. Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package) cmake/torch.cmake:14 (find_package) CMakeLists.txt:55 (include) This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda10.2/toolkit/10.2.89 (found version "10.2") -- Caffe2: CUDA detected: 10.2 -- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc -- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89 -- Caffe2: Header version is: 10.2 -- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) CMake Warning at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:109 (message): Caffe2: Cannot find cuDNN library. Turning the option off Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package) cmake/torch.cmake:14 (find_package) CMakeLists.txt:55 (include)

-- Autodetected CUDA architecture(s): 6.0 -- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60 CMake Error at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message): Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN libraries. Please set the proper cuDNN prefixes and / or install cuDNN. Call Stack (most recent call first): /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package) cmake/torch.cmake:14 (find_package) CMakeLists.txt:55 (include)

-- Configuring incomplete, errors occurred! See also "/alt-arabic/speech/amir/k2/tmp/kaldifeat/build/CMakeFiles/CMakeOutput.log". See also "/alt-arabic/speech/amir/k2/tmp/kaldifeat/build/CMakeFiles/CMakeError.log".

csukuangfj commented 1 year ago

Please use

wget https://huggingface.co/csukuangfj/cudnn/resolve/main/cudnn-10.2-linux-x64-v8.0.2.39.tgz
tar xvf cudnn-10.2-linux-x64-v8.0.2.39.tgz --strip-components=1 -C /cm/shared/apps/cuda10.2/toolkit/10.2.89

to install cuDNN and retry

cd /path/to/kaldifeat
rm -rf build
mkdir build
cd build
cmake ..
make
AmirHussein96 commented 1 year ago

Thanks @csukuangfj it worked with the following:

cmake -DCMAKE_BUILD_TYPE=Release -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so
 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include ..

make -j8
AmirHussein96 commented 1 year ago

@csukuangfj How can I export the library to use it. I tried the following but it did not work

kfeat=/alt-arabic/speech/amir/k2/tmp/kaldifeat/
export PYTHONPATH=$kfeat/kaldifeat/python:$PYTHONPATH
export PYTHONPATH=$kfeat/kaldifeat/build/lib:$PYTHONPATH
csukuangfj commented 1 year ago

Could you show the error logs?

AmirHussein96 commented 1 year ago

@csukuangfj there is no error, I managed to build kaldifeat from source using cmake, and make succesfully. But when I import it in python it does not work, so I though it might be similar to k2 that I need to export python path right?

csukuangfj commented 1 year ago

But when I import it in python it does not work,

If it does not work, it should give you some message indicating it is not working.

so I though it might be similar to k2 that I need to export python path right

Yes, if you use

git clone https://github.com/csukuangfj/kaldifeat
cd kaldifeat
mkdir build
cd build
cmake ..
make -j

Then you have to use

export PYTHONPATH=/path/to/kaldifeat/build/lib:$PYTHONPATH
export PYTHONPATH=/path/to/kaldifeat/kaldifeat/python:$PYTHONPATH
AmirHussein96 commented 1 year ago

Thanks it worked, you can close the issue.