search

k2-fsa / k2

FSA/FST algorithms, differentiable, with PyTorch compatibility.
https://k2-fsa.github.io/k2
Apache License 2.0
1.11k stars 213 forks source link

Compilation error with `CUDA_cublas_LIBRARY` #1178

Closed cantabile-kwok closed 1 year ago

cantabile-kwok commented 1 year ago

I am compiling k2 from source on a linux server, and I encountered some problems while compilation. Here is my environment description:

OS: CentOS 7 Linux
python: Python 3.10.11 (in conda environment)
torch: 1.11.0+cu102
CUDA: 10.2
cmake: 3.12.0-rc2
gcc: 8.4.0

I installed CuDNN, cudatoolkit, and cublas from conda (is it not going to work?)

I manually specified CUDNN_LIBRARY_PATH in CMakeCache.txt and the cmake configuration step succeeds. But in the compilation process, errors occurred. The following

/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing k2.egg-info/PKG-INFO
writing dependency_links to k2.egg-info/dependency_links.txt
writing requirements to k2.egg-info/requires.txt
writing top-level names to k2.egg-info/top_level.txt
reading manifest file 'k2.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'k2.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying k2/python/k2/__init__.py -> build/lib.linux-x86_64-cpython-310/k2
running build_ext
-- CMAKE_VERSION: 3.12.0-rc2
-- Enabled languages: CXX;CUDA
-- K2_OS: 
-- C++ Standard version: 14
-- Autodetected CUDA architecture(s):  7.0
-- K2_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70
-- K2_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75
-- Skipping arch 35
-- Skipping arch 50
-- Skipping arch 60
-- Skipping arch 61
-- Adding arch 70
-- Skipping arch 75
-- K2_COMPUTE_ARCHS: 70
-- Could NOT find Valgrind (missing: Valgrind_INCLUDE_DIR Valgrind_EXECUTABLE) 
-- Downloading pybind11 from file:///mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/pybind11-5bc0943ed96836f46489f53961f6c438d2935357.zip
-- pybind11 is downloaded to /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/pybind11-src
-- pybind11 v2.11.0 dev1
-- Python executable: /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/bin/python3
-- Caffe2: CUDA detected: 10.2
-- Caffe2: CUDA nvcc is: /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/bin/nvcc
-- Caffe2: CUDA toolkit directory: /mnt/lustre/cm/shared/global/src/dev/cuda/10.2
-- Caffe2: Header version is: 10.2
-- Found cuDNN: v7.6.5  (include: /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/include, library: /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/libcudnn.so.7.6.5)
-- /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64/libnvrtc.so shorthash is 08c4863f
-- Autodetected CUDA architecture(s):  7.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
CMake Warning at /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  cmake/torch.cmake:11 (find_package)
  CMakeLists.txt:292 (include)

-- K2_TORCH_VERSION: 1.11
-- PyTorch version: 1.11.0+cu102
-- PyTorch cuda version: 10.2
-- Generated /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/torch_version.py
-- Downloading cub from file:///mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/cub-1.15.0.tar.gz
-- cub is downloaded to /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/cub-src
-- Downloading moderngpu from file:///mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/moderngpu-8ec9ac0de8672de7217d014917eedec5317f75f3.zip
-- moderngpu is downloaded to /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/moderngpu-src
-- CMAKE_CUDA_FLAGS:   -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w  --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_70,code=sm_70 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall  --compiler-options -Wno-strict-overflow  --compiler-options -Wno-unknown-pragmas 
-- CMAKE_CXX_FLAGS:  -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable  -Wno-strict-overflow 
-- CMAKE_CUDA_FLAGS:   -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w  --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_70,code=sm_70 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall  --compiler-options -Wno-strict-overflow  --compiler-options -Wno-unknown-pragmas 
-- Generated /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/k2/csrc/version.h
-- PYTHON_LIBRARY: /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/libpython3.10.so
-- Including k2/torch. K2_TORCH_VERSION is 1.11
-- Downloading kaldifeat from file:///mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/kaldifeat-1.20.tar.gz
-- kaldifeat is downloaded to /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src
-- kaldifeat's binary dir is /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-build
-- C++ Standard version: 14
-- Python executable: /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/bin/python3
CMake Warning at /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  cmake/torch.cmake:11 (find_package)
  build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/CMakeLists.txt:55 (include)

-- K2_TORCH_VERSION: 1.11
-- PyTorch version: 1.11.0+cu102
-- PyTorch cuda version: 10.2
-- CMAKE_CXX_FLAGS:  -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable  -Wno-strict-overflow  -D_GLIBCXX_USE_CXX11_ABI=0
-- CMAKE_INSTALL_PREFIX: /mnt/lustre/xxx/home/ywg12/tools/k2/build/lib.linux-x86_64-cpython-310/k2
-- All headers: /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-common-inl.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-common.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-fbank.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-functions.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-mfcc.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-plp.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-spectrogram.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/feature-window.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/log.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/matrix-functions.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/mel-computations.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/online-feature-itf.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/online-feature.h;/mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/pitch-functions.h
-- Configuring done
CMake Warning at k2/csrc/CMakeLists.txt:101 (add_library):
  Cannot generate a safe runtime search path for target context because files
  in some directories may conflict with libraries in implicit directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at build/temp.linux-x86_64-cpython-310/_deps/pybind11-src/tools/pybind11Tools.cmake:177 (add_library):
  Cannot generate a safe runtime search path for target _k2 because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  k2/python/csrc/CMakeLists.txt:32 (pybind11_add_module)

CMake Warning at build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/CMakeLists.txt:31 (add_executable):
  Cannot generate a safe runtime search path for target test_kaldifeat
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/csrc/CMakeLists.txt:15 (add_library):
  Cannot generate a safe runtime search path for target kaldifeat_core
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at build/temp.linux-x86_64-cpython-310/_deps/pybind11-src/tools/pybind11Tools.cmake:177 (add_library):
  Cannot generate a safe runtime search path for target _kaldifeat because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  build/temp.linux-x86_64-cpython-310/_deps/kaldifeat-src/kaldifeat/python/csrc/CMakeLists.txt:2 (pybind11_add_module)

CMake Warning at k2/torch/csrc/CMakeLists.txt:73 (add_library):
  Cannot generate a safe runtime search path for target k2_torch_api because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/csrc/CMakeLists.txt:27 (add_library):
  Cannot generate a safe runtime search path for target k2_fbank because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/csrc/CMakeLists.txt:24 (add_library):
  Cannot generate a safe runtime search path for target k2_torch because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:17 (add_executable):
  Cannot generate a safe runtime search path for target ctc_decode because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:28 (add_executable):
  Cannot generate a safe runtime search path for target hlg_decode because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:39 (add_executable):
  Cannot generate a safe runtime search path for target ngram_lm_rescore
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:50 (add_executable):
  Cannot generate a safe runtime search path for target attention_rescore
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:63 (add_executable):
  Cannot generate a safe runtime search path for target online_decode because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:75 (add_executable):
  Cannot generate a safe runtime search path for target rnnt_demo because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

CMake Warning at k2/torch/bin/CMakeLists.txt:87 (add_executable):
  Cannot generate a safe runtime search path for target
  pruned_stateless_transducer because files in some directories may conflict
  with libraries in implicit directories:

    runtime library [libnvrtc.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcudart.so.10.2] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcufft.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib
    runtime library [libcurand.so.10] in /mnt/lustre/cm/shared/global/src/dev/cuda/10.2/lib64 may be hidden by files in:
      /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /mnt/lustre/xxx/home/ywg12/tools/k2/build/temp.linux-x86_64-cpython-310
/**
 * @brief
 * This file contains information for building k2, which is helpful
 * for describing the environment in which k2 is used when creating
 * an issue on GitHub.
 *
 * @copyright
 * Copyright (c)  2020  Mobvoi Inc.        (authors: Fangjun Kuang)
 *
 * @copyright
 * See LICENSE for clarification regarding multiple authors
 *
 * @caution
 * This file is auto-generated by CMake. Do NOT edit!
 * You can find it at build_dir/k2/csrc/version.h
 */

#ifndef K2_CSRC_VERSION_H_
#define K2_CSRC_VERSION_H_

namespace k2 {

// Version of k2 in "major.minor.patch" format
static constexpr const char *kVersion = "1.23.4";

// The commit used to build k2
static constexpr const char *kGitSha1 = "";

// Date of the commit used to build k2
static constexpr const char *kGitDate = "";

// Version of CUDA used to build k2.
// Its format is "major.minor", e.g., 10.1
static constexpr const char *kCudaVersion = "10.2";

// cuDNN version, e.g., 8.0.2
static constexpr const char *kCudnnVersion = "7.6.5";

// clang-format off
// Version of Python used to build k2 Python bindings.
static constexpr const char *kPythonVersion = "3.10";

// clang-format on

// CMake build type, e.g., Release or Debug.
static constexpr const char *kBuildType = "Release";

// The operating system that is used to build k2, e.g., Ubuntu 16.04 LTS
static constexpr const char *kOS = R"os()os";

// e.g., 3.18.0
static constexpr const char *kCMakeVersion = "3.12.0-rc2";

// Version of the compiler, e.g., 5.4.0
static constexpr const char *kGCCVersion = "8.4.0";

// CUDA flags used to compile k2
static constexpr const char *kCMakeCudaFlags = R"cuda_flags(  -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w  --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_70,code=sm_70 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall  --compiler-options -Wno-strict-overflow  --compiler-options -Wno-unknown-pragmas )cuda_flags";

// CXX flags used to compile k2
static constexpr const char *kCMakeCxxFlags = R"cxx_flags( -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable  -Wno-strict-overflow )cxx_flags";

// Which PyTorch version k2 is using, e.g., 1.6.0+cu101
static constexpr const char *kTorchVersion = "1.11.0+cu102";

// Which CUDA version PyTorch is using, e.g., 10.1
static constexpr const char *kTorchCudaVersion = "10.2";

#ifndef K2_WITH_CUDA
#define K2_WITH_CUDA
#endif

#ifdef K2_WITH_CUDA
  static constexpr bool kWithCuda = true;
#else
  static constexpr bool kWithCuda = false;
#endif

// Indicate whether NVTX is enabled or not
#ifndef K2_ENABLE_NVTX
#define K2_ENABLE_NVTX
#endif

#ifdef K2_ENABLE_NVTX
static constexpr bool kEnableNvtx = true;
#else
static constexpr bool kEnableNvtx = false;
#endif
}  // namespace k2

#endif  //  K2_CSRC_VERSION_H_
[  2%] Built target k2_log
[ 15%] Built target fsa
[ 15%] Linking CUDA device code CMakeFiles/context.dir/cmake_device_link.o
nvcc fatal   : Unknown option '-Wl,--no-as-needed,/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so'
make[2]: *** [k2/csrc/CMakeFiles/context.dir/cmake_device_link.o] Error 1
make[1]: *** [k2/csrc/CMakeFiles/context.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 16%] Linking CXX shared library ../../../../lib/libkaldifeat_core.so
/usr/bin/ld: cannot find -lCUDA_cublas_LIBRARY-NOTFOUND
collect2: error: ld returned 1 exit status
make[2]: *** [lib/libkaldifeat_core.so] Error 1
make[1]: *** [_deps/kaldifeat-build/kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/all] Error 2
make: *** [all] Error 2
cmake_path: /mnt/lustre/cm/shared/global/src/dev/cmake/3.12.0/bin/cmake
Setting PYTHON_EXECUTABLE to /mnt/lustre/xxx/home/ywg12/.conda/envs/k2/bin/python3
build command is:

                cd build/temp.linux-x86_64-cpython-310

                cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/bin/python3 -DK2_ENABLE_BENCHMARK=OFF  -DK2_ENABLE_TESTS=OFF  -DCMAKE_INSTALL_PREFIX=/mnt/lustre/xxx/home/ywg12/tools/k2/build/lib.linux-x86_64-cpython-310/k2  /mnt/lustre/xxx/home/ywg12/tools/k2

                cat k2/csrc/version.h

                make -j6 install

Traceback (most recent call last):
  File "/mnt/lustre/xxx/home/ywg12/tools/k2/setup.py", line 235, in <module>
    setuptools.setup(
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
    self._build_extensions_serial()
  File "/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
    self.build_extension(ext)
  File "/mnt/lustre/xxx/home/ywg12/tools/k2/setup.py", line 207, in build_extension
    raise Exception("Failed to build k2")
Exception: Failed to build k2

Note that I also specified CUDA_cublas_LIBRARY:FILEPATH=/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/libcublas.so in the CMakeCache.txt, but I found this file is overwritten by CUDA_cublas_LIBRARY:FILEPATH=CUDA_cublas_LIBRARY-NOTFOUND the next time I run python3 setup.py install. As a result, the program fails to locate the cublas library which causes the error.

But I don't know how to provide this cublas path to the program. Also, I notice that there are a lot of lines reporting Some of these libraries may not be found correctly. Could anyone figure out what is going wrong please? 🙏

csukuangfj commented 1 year ago

cmake: 3.12.0-rc2

nvcc fatal : Unknown option '-Wl,--no-as-needed,/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so'

Please switch to another version of cmake,e.g., cmake 3.18.

Note that I also specified CUDA_cublas_LIBRARY:FILEPATH=/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/libcublas.so in the CMakeCache.txt, but I found this file is overwritten by CUDA_cublas_LIBRARY:FILEPATH=CUDA_cublas_LIBRARY-NOTFOUND the next time I run python3 setup.py install.

Please don't change CMakeCache.txt. Please use the following commands

rm -rf build

export K2_CMAKE_ARGS="-DCUDA_cublas_LIBRARY=/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/libcublas.so -DCUDNN_INCLUDE_PATH=/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/include -DCUDNN_LIBRARY_PATH=/mnt/lustre/xxx/home/ywg12/.conda/envs/k2/lib/libcudnn.so.7.6.5"

python3 setup.py install

Also, I notice that there are a lot of lines reporting Some of these libraries may not be found correctly.

Please ignore those warnings for now.

By the way, if you follow https://k2-fsa.github.io/k2/installation/cuda-cudnn.html to install cudatoolkit and cuDNN and use pip install to replace conda install, you won't get such warnings.

cantabile-kwok commented 1 year ago

This is very very helpful 👍 The installation is successful now! Much thanks!