SJTU-Thinklab-Det / r3det-pytorch

R3Det based on mmdet 2.19.0
Apache License 2.0
42 stars 7 forks source link

pip install -v -e . #9

Closed houw0517 closed 2 years ago

houw0517 commented 2 years ago
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/4] /usr/local/cuda-11.0/bin/nvcc -DWITH_CUDA -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/TH -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/hw/anaconda3/envs/R3Det/include/python3.8 -c -c /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/nms_rotated_cuda.cu -o /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=nms_rotated_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
FAILED: /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_cuda.o
/usr/local/cuda-11.0/bin/nvcc -DWITH_CUDA -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/TH -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/hw/anaconda3/envs/R3Det/include/python3.8 -c -c /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/nms_rotated_cuda.cu -o /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=nms_rotated_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
nvcc fatal   : Unsupported gpu architecture 'compute_86'
[2/4] /usr/local/cuda-11.0/bin/nvcc -DWITH_CUDA -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/TH -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/hw/anaconda3/envs/R3Det/include/python3.8 -c -c /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/poly_nms_cuda.cu -o /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/poly_nms_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=nms_rotated_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
FAILED: /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/poly_nms_cuda.o
/usr/local/cuda-11.0/bin/nvcc -DWITH_CUDA -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/TH -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/hw/anaconda3/envs/R3Det/include/python3.8 -c -c /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/poly_nms_cuda.cu -o /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/poly_nms_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=nms_rotated_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
nvcc fatal   : Unsupported gpu architecture 'compute_86'
[3/4] c++ -MMD -MF /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_cpu.o.d -pthread -B /home/hw/anaconda3/envs/R3Det/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/TH -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/hw/anaconda3/envs/R3Det/include/python3.8 -c -c /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/nms_rotated_cpu.cpp -o /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=nms_rotated_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
[4/4] c++ -MMD -MF /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_ext.o.d -pthread -B /home/hw/anaconda3/envs/R3Det/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/TH -I/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/hw/anaconda3/envs/R3Det/include/python3.8 -c -c /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/nms_rotated_ext.cpp -o /home/hw/Project/r3det-pytorch/build/temp.linux-x86_64-3.8/r3det/ops/nms_rotated/src/nms_rotated_ext.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=nms_rotated_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:149,
                 from /home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                 from /home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                 from /home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                 from /home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                 from /home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from /home/hw/Project/r3det-pytorch/r3det/ops/nms_rotated/src/nms_rotated_ext.cpp:5:
/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
   84 | #pragma omp parallel for if ((end - begin) >= grain_size)
      |
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1533, in _run_ninja_build
    subprocess.run(
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/hw/Project/r3det-pytorch/setup.py", line 178, in <module>
    setup(
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/setuptools/command/develop.py", line 114, in install_for_development
    self.run_command('build_ext')
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 670, in build_extensions
    build_ext.build_extensions(self)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 202, in build_extension
    _build_ext.build_extension(self, ext)
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 491, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1250, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/home/hw/anaconda3/envs/R3Det/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1555, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

ERROR: Command errored out with exit status 1: /home/hw/anaconda3/envs/R3Det/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/hw/Project/r3det-pytorch/setup.py'"'"'; file='"'"'/home/hw/Project/r3det-pytorch/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

zytx121 commented 2 years ago

This means that the compiler should optimize for sm_86, i.e., Nvidia 30 series card, but such optimizations have not been supported by CUDA toolkit 11.0. This work-around modifies the compile flag by adding export TORCH_CUDA_ARCH_LIST="7.5" This may hurt the performance but it works.

zytx121 commented 2 years ago

https://github.com/WongKinYiu/ScaledYOLOv4/issues/119

houw0517 commented 2 years ago

After export TORCH_CUDA_ARCH_LIST="7.5" It also appears that RuntimeError: CUDA error: no kernel image is available for execution on the device

zytx121 commented 2 years ago

After export TORCH_CUDA_ARCH_LIST="7.5" It also appears that RuntimeError: CUDA error: no kernel image is available for execution on the device

This is our environment when using Nvidia 3090:

torcch 1.8.1+cu111
torchvision 0.9.1+cu111
mmcv 1.3.8 (compile by myself on 3090)
mmdet 2.18.0
cudatoolkit 11.1

https://github.com/open-mmlab/mmdetection/blob/5e246d5e3bc3310b5c625fb57bc03d2338ca39bc/docs/en/faq.md

houw0517 commented 2 years ago

This means that the compiler should optimize for sm_86, i.e., Nvidia 30 series card, but such optimizations have not been supported by CUDA toolkit 11.0. This work-around modifies the compile flag by adding export TORCH_CUDA_ARCH_LIST="7.5" This may hurt the performance but it works.

请问大佬 cuda11.0不支持算力8.6 cuda11.1支持吗

houw0517 commented 2 years ago

After export TORCH_CUDA_ARCH_LIST="7.5" It also appears that RuntimeError: CUDA error: no kernel image is available for execution on the device

This is our environment when using Nvidia 3090:

torcch 1.8.1+cu111
torchvision 0.9.1+cu111
mmcv 1.3.8 (compile by myself on 3090)
mmdet 2.18.0
cudatoolkit 11.1

https://github.com/open-mmlab/mmdetection/blob/5e246d5e3bc3310b5c625fb57bc03d2338ca39bc/docs/en/faq.md

请问大佬 这一步 mmcv 1.3.8 (compile by myself on 3090) 是什么意思啊? RuntimeError: CUDA error: no kernel image is available for execution on the device 这个报错还是不知道怎么解决