Closed a897456 closed 1 month ago
Can you please try what ChatGPT suggests:
The error you're encountering seems to be related to building a CUDA extension, specifically the upfirdn2d
kernel, using the Ninja build system. The error message indicates that the process failed when running the nvcc
compiler. Here are a few steps you can take to try and resolve this issue:
CUDA Version Compatibility: Ensure that the version of CUDA you are using (v12.1) is compatible with the version of PyTorch you have installed and the compute capability of your GPU.
Check CUDA and NVIDIA Driver:
CUDA_HOME
, PATH
, and LD_LIBRARY_PATH
) are correctly set.Reinstall PyTorch with CUDA support: If there are compatibility issues, consider reinstalling PyTorch with the appropriate CUDA version. You can find the right command for your setup on the https://pytorch.org/get-started/locally/">PyTorch website.
Install Ninja: Ensure that Ninja build system is installed and accessible in your PATH. You can install it via Conda:
sh conda install -c conda-forge ninja
Compile Manually:
Sometimes building the extension manually can provide more diagnostic information. Navigate to the directory containing the upfirdn2d_kernel.cu
file and run the nvcc
command manually to see if there are more detailed errors.
Modify ARCH_LIST:
ARCH_LIST
is set to 8.9
which corresponds to NVIDIA Ada GPUs like the RTX 40 series. If you're using a different GPU, ensure TORCH_CUDA_ARCH_LIST
matches your GPU architecture. You can find the appropriate architecture version https://developer.nvidia.com/cuda-gpus">here.
Simplified Compilation Command:
Simplify your nvcc
command to isolate issues. For example:
sh nvcc -arch=sm_89 --expt-relaxed-constexpr -std=c++17 -c -o upfirdn2d_kernel.cuda.o upfirdn2d_kernel.cu
Here's a basic checklist to follow:
sh conda install pytorch torchvision torchaudio cudatoolkit=12.1 -c pytorch
python import torch print(torch.cuda.is_available())
sh set TORCH_CUDA_ARCH_LIST=8.9
sh conda install ninja
Here's an example of checking the CUDA versions and toolchain: sh nvcc --version
Verify it matches the expected version and ensure your GPU driver is recent.
Please apply these steps and let me know if the issue persists!
E:\ProgramData\anaconda3\envs\py39sgmse\python.exe E:\000\sgmse-main\train.py --base_dir E:/000/sgmse-main/Data_Librispeech Set TORCH_CUDA_ARCH_LIST to: 8.9 Traceback (most recent call last): File "E:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\utils\cpp_extension.py", line 2107, in _run_ninja_build subprocess.run( File "E:\ProgramData\anaconda3\envs\py39sgmse\lib\subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "E:\000\sgmse-main\train.py", line 14, in
from sgmse.backbones.shared import BackboneRegistry
File "E:\000\sgmse-main\sgmse\backbones__init.py", line 2, in
from .ncsnpp import NCSNpp
File "E:\000\sgmse-main\sgmse\backbones\ncsnpp.py", line 18, in
from .ncsnpp_utils import layers, layerspp, normalization
File "E:\000\sgmse-main\sgmse\backbones\ncsnpp_utils\layerspp.py", line 20, in
from . import up_or_down_sampling
File "E:\000\sgmse-main\sgmse\backbones\ncsnpp_utils\up_or_down_sampling.py", line 10, in
from .op import upfirdn2d
File "E:\000\sgmse-main\sgmse\backbones\ncsnpp_utils\op__init__.py", line 1, in
from .upfirdn2d import upfirdn2d
File "E:\000\sgmse-main\sgmse\backbones\ncsnpp_utils\op\upfirdn2d.py", line 12, in
upfirdn2d_op = load(
File "E:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\utils\cpp_extension.py", line 1309, in load
return _jit_compile(
File "E:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\utils\cpp_extension.py", line 1719, in _jit_compile
_write_ninja_file_and_build_library(
File "E:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\utils\cpp_extension.py", line 1832, in _write_ninja_file_and_build_library
_run_ninja_build(
File "E:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\utils\cpp_extension.py", line 2123, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'upfirdn2d': [1/2] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d_kernel.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=upfirdn2d -DTORCH_API_INCLUDE_EXTENSION_H -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include\torch\csrc\api\include -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include\TH -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -IE:\ProgramData\anaconda3\envs\py39sgmse\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D CUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_89,code=sm_89 -std=c++17 -c E:\000\sgmse-main\sgmse\backbones\ncsnpp_utils\op\upfirdn2d_kernel.cu -o upfirdn2d_kernel.cuda.o
FAILED: upfirdn2d_kernel.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d_kernel.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=upfirdn2d -DTORCH_API_INCLUDE_EXTENSION_H -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include\torch\csrc\api\include -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include\TH -IE:\ProgramData\anaconda3\envs\py39sgmse\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -IE:\ProgramData\anaconda3\envs\py39sgmse\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=sm_89 -std=c++17 -c E:\000\sgmse-main\sgmse\backbones\ncsnpp_utils\op\upfirdn2d_kernel.cu -o upfirdn2d_kernel.cuda.o
ninja: build stopped: subcommand failed.