"ModuleNotFoundError: No module named '_optimized_transducer'" when testing.

[luomingshuang]

I install the optimized_transducer as follows:

git clone https://github.com/csukuangfj/optimized_transducer.git
cd optimized_transducer
mkdir build
cd build
cmake -DOT_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release ..
export PYTHONPATH=$PWD/../optimized_transducer/python:$PWD/lib:$PYTHONPATH

The cmake log as follows:

-- Enabled languages: CXX;CUDA
-- The CXX compiler identification is GNU 7.5.0
-- The CUDA compiler identification is NVIDIA 10.1.243
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Autodetected CUDA architecture(s):  7.0
-- OT_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70
-- OT_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75
-- Skipping arch 35
-- Skipping arch 50
-- Skipping arch 60
-- Skipping arch 61
-- Adding arch 70
-- Skipping arch 75
-- OT_COMPUTE_ARCHS: 70
-- Downloading pybind11
-- pybind11 is downloaded to /ceph-meixu/luomingshuang/optimized_transducer/build/_deps/pybind11-src
-- pybind11 v2.6.0
-- Found PythonInterp: /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/bin/python (found version "3.8.11")
-- Found PythonLibs: /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/bin/python
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "10.1")
-- Caffe2: CUDA detected: 10.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 10.1
-- Found CUDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found cuDNN: v7.6.2  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Autodetected CUDA architecture(s):  7.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
-- Found Torch: /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/lib/libtorch.so
-- PyTorch version: 1.7.0+cu101
-- PyTorch cuda version: 10.1
-- Use FetchContent provided by k2
-- Downloading googletest

-- googletest is downloaded to /ceph-meixu/luomingshuang/optimized_transducer/build/_deps/googletest-src
-- googletest's binary dir is /ceph-meixu/luomingshuang/optimized_transducer/build/_deps/googletest-build
-- The C compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Downloading moderngpu
-- moderngpu is downloaded to /ceph-meixu/luomingshuang/optimized_transducer/build/_deps/moderngpu-src
-- Configuring done
-- Generating done
-- Build files have been written to: /ceph-meixu/luomingshuang/optimized_transducer/build

But when I use python optimized_transducer/python/tests/test_compute_transducer_loss.py for testing, there is an error as follows:

/ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torchaudio/backend/utils.py:53: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.
  warnings.warn(
Traceback (most recent call last):
  File "optimized_transducer/python/tests/test_compute_transducer_loss.py", line 8, in <module>
    import optimized_transducer
  File "/ceph-meixu/luomingshuang/optimized_transducer/optimized_transducer/python/optimized_transducer/__init__.py", line 1, in <module>
    from .transducer_loss import TransducerLoss, transducer_loss  # noqa
  File "/ceph-meixu/luomingshuang/optimized_transducer/optimized_transducer/python/optimized_transducer/transducer_loss.py", line 3, in <module>
    import _optimized_transducer
ModuleNotFoundError: No module named '_optimized_transducer'

Hope to know how I can solve it. Thanks!

[csukuangfj]

Please do

cd build
make -j

You have to run make after running cmake.

[luomingshuang]

Ok. But it seems there are still some errors in the make -j process. The part of log which records the errors as follows:

luomingshuang@de-74279-k2-train-2-0815224919-75d558775b-mmnv8:/ceph-meixu/luomingshuang/optimized_transducer/build$ make -j
Scanning dependencies of target optimized_transducer_core_cuda
[ 22%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/cuda.cu.o
[ 22%] Building CUDA object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core_cuda.dir/kernels.cu.o
[ 33%] Linking CUDA device code CMakeFiles/optimized_transducer_core_cuda.dir/cmake_device_link.o
[ 44%] Linking CUDA shared library ../../lib/liboptimized_transducer_core_cuda.so
[ 44%] Built target optimized_transducer_core_cuda
Scanning dependencies of target optimized_transducer_core
[ 66%] Building CXX object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core.dir/transducer-loss.cc.o
[ 66%] Building CXX object optimized_transducer/csrc/CMakeFiles/optimized_transducer_core.dir/cpu.cc.o
[ 77%] Linking CXX shared library ../../lib/liboptimized_transducer_core.so
[ 77%] Built target optimized_transducer_core
Scanning dependencies of target _optimized_transducer
[ 88%] Building CXX object optimized_transducer/python/csrc/CMakeFiles/_optimized_transducer.dir/optimized_transducer.cc.o
In file included from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3:0,
                 from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/torch/script.h:3,
                 from /ceph-meixu/luomingshuang/optimized_transducer/optimized_transducer/csrc/transducer-loss.h:7,
                 from /ceph-meixu/luomingshuang/optimized_transducer/optimized_transducer/python/csrc/optimized_transducer.cc:7:
/ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/ATen/ATen.h:4:2: error: #error C++14 or later compatible compiler is required to use ATen.
 #error C++14 or later compatible compiler is required to use ATen.
  ^~~~~
In file included from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/c10/util/ArrayRef.h:19:0,
                 from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/c10/core/DispatchKey.h:7,
                 from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/c10/core/Backend.h:4,
                 from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/c10/core/Layout.h:3,
                 from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:4,
                 from /ceph-meixu/luomingshuang/anaconda3/envs/k2-python/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3,
.......

So should