Vectorrent
commented
1 year ago Same issue here, when trying to compile inside of the official NVIDIA Docker image:
=> ERROR [vtx 15/16] RUN python3 /vtx/lab/rwkv-cuda/wkv/run.py 20.2s
------
> [vtx 15/16] RUN python3 /vtx/lab/rwkv-cuda/wkv/run.py:
#0 1.852 No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
#0 1.853 Using /root/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
#0 1.853 Creating extension directory /root/.cache/torch_extensions/py310_cu117/wkv...
#0 1.905 Detected CUDA files, patching ldflags
#0 1.905 Emitting ninja build file /root/.cache/torch_extensions/py310_cu117/wkv/build.ninja...
#0 1.905 Building extension module wkv...
#0 1.905 Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
#0 2.947 [1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math --extra-device-vectorization -std=c++14 -c /vtx/lab/rwkv-cuda/wkv/cuda/wkv_cuda_v2.cu -o wkv_cuda_v2.cuda.o
#0 19.14 [2/3] c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 /wd4624 -c /vtx/lab/rwkv-cuda/wkv/cuda/wkv_op.cpp -o wkv_op.o
#0 19.14 FAILED: wkv_op.o
#0 19.14 c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 /wd4624 -c /vtx/lab/rwkv-cuda/wkv/cuda/wkv_op.cpp -o wkv_op.o
#0 19.14 c++: warning: /wd4624: linker input file unused because linking not done
#0 19.14 c++: error: /wd4624: linker input file not found: No such file or directory
#0 19.14 ninja: build stopped: subcommand failed.
#0 19.17 Traceback (most recent call last):
#0 19.17 File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
#0 19.17 subprocess.run(
#0 19.17 File "/usr/lib/python3.10/subprocess.py", line 524, in run
#0 19.18 raise CalledProcessError(retcode, process.args,
#0 19.18 subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
#0 19.18
#0 19.18 The above exception was the direct cause of the following exception:
#0 19.18
#0 19.18 Traceback (most recent call last):
#0 19.18 File "/vtx/lab/rwkv-cuda/wkv/run.py", line 86, in <module>
#0 19.18 wkv_cuda = load(name="wkv", sources=["cuda/wkv_op.cpp", f"cuda/wkv_cuda_v{CUDA_KERNEL_VERSION}.cu"],
#0 19.18 File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1284, in load
#0 19.18 return _jit_compile(
#0 19.18 File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
#0 19.18 _write_ninja_file_and_build_library(
#0 19.18 File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
#0 19.18 _run_ninja_build(
#0 19.18 File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
#0 19.18 raise RuntimeError(message) from e
#0 19.18 RuntimeError: Error building extension 'wkv'
------
failed to solve: process "/bin/sh -c python3 /vtx/lab/rwkv-cuda/wkv/run.py" did not complete successfully: exit code: 1
Chengyui
(gh_baize-chatbot) ub2004@ub2004-B85M-A0:~/llm_dev/RWKV-CUDA/wkv$ python3 run.py Using /home/ub2004/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Creating extension directory /home/ub2004/.cache/torch_extensions/py38_cu117/wkv... Detected CUDA files, patching ldflags Emitting ninja build file /home/ub2004/.cache/torch_extensions/py38_cu117/wkv/build.ninja... Building extension module wkv... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 /wd4624 -c /home/ub2004/llm_dev/RWKV-CUDA/wkv/cuda/wkv_op.cpp -o wkv_op.o FAILED: wkv_op.o c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 /wd4624 -c /home/ub2004/llm_dev/RWKV-CUDA/wkv/cuda/wkv_op.cpp -o wkv_op.o c++: error: /wd4624: No such file or directory [2/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' --use_fast_math --extra-device-vectorization -std=c++17 -c /home/ub2004/llm_dev/RWKV-CUDA/wkv/cuda/wkv_cuda_v2.cu -o wkv_cuda_v2.cuda.o ninja: build stopped: subcommand failed. Traceback (most recent call last): File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build subprocess.run( File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "run.py", line 86, in
wkv_cuda = load(name="wkv", sources=["cuda/wkv_op.cpp", f"cuda/wkv_cuda_v{CUDA_KERNEL_VERSION}.cu"],
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'wkv'
(gh_baize-chatbot) ub2004@ub2004-B85M-A0:~/llm_dev/RWKV-CUDA/wkv$