farleylai
commented
5 years ago It seems like for some reason, the default nvcc compilation does not generate code support for GTX 1080 Ti on a Titan X machine.
In that case, one may explicitly add the architecture and gencode options in the setup.py.
Here is the example of mmdet/ops/nms/setup.py:
nvcc_ARCH = ['-arch=sm_52']
nvcc_ARCH += ["-gencode=arch=compute_75,code=\"compute_75\""]
nvcc_ARCH += ["-gencode=arch=compute_75,code=\"sm_75\""]
nvcc_ARCH += ["-gencode=arch=compute_70,code=\"sm_70\""]
nvcc_ARCH += ["-gencode=arch=compute_61,code=\"sm_61\""]
nvcc_ARCH += ["-gencode=arch=compute_52,code=\"sm_52\""]
extra_compile_args = {
'cxx': ['-Wno-unused-function', '-Wno-write-strings'],
'nvcc': nvcc_ARCH,}Then specify the extra_compile_args composed from the above:
setup(
name='nms_cuda',
ext_modules=[
CUDAExtension('nms_cuda', [
'src/nms_cuda.cpp',
'src/nms_kernel.cu',
],
extra_compile_args=extra_compile_args,
),
CUDAExtension('nms_cpu', [
'src/nms_cpu.cpp',
]),
],
cmdclass={'build_ext': BuildExtension})By rebuilding this particular nms on the TITAN X machine and reinstalling mmdet, it now works on GTX 1080 Ti too.
Nonetheless, the real cause is likely something behind.
Since the setup.py is per CUDA module basis, it would be tedious to change all.
Any better suggestion and clarifications?
Update: it should be Titan X, not V.
hellock
Simon4Yan
MikhailMashukov
juanluisrosaramos
The error is thrown when
mmdetis compiled on a user machine with Titan V (Pascal) and executes on a cluster worker machine with a newer GTX 1080 Ti.Both machines have CUDA 10 installed. However, if compiled on a machine with RTX Titan, the execution on both machines with Titan V and GTX 1080 Ti is fine. After some cross-testing, it seems like this is the only failure case:
Any idea to address this failure case?
Update: it should be Titan X, not V.
Here is the minimal code to reproduce with GPU nms():
The error:
PyTorch 1.1 from miniconda 3