BVLC / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
34.14k stars 18.68k forks source link

nvcc fatal : Unsupported gpu architecture 'compute_61' #5141

Closed soulslicer closed 7 years ago

soulslicer commented 7 years ago

Issue summary

I have the following configuration below but at the very start of compilation, it throws the errors for all gpu modules:

nvcc fatal : Unsupported gpu architecture 'compute_61'

Why is this the case? Is 7.5 not supported on the master branch for the new Geforce 1070/1080 cards?

Your system configuration

Operating system: Compiler: GCC 4.8.4 CUDA version (if applicable): 7.5 CUDNN version (if applicable): 5.0 BLAS: Python or MATLAB version (for pycaffe and matcaffe respectively):

SIshijima commented 7 years ago

CUDA 7.5 is old so it may not support 'compute_61'. CUDA8.0 or selecting 'compute_50' in Make.config will be a solution, maybe.

I'm using CUDA8.0 + Geforce1070 with no problem.

soulslicer commented 7 years ago

Yeah..it would be nice to have release branches though..instead of a master branch that you never know might break with a new opencv or cuda or whatever release

microljy commented 7 years ago

You should modify Makefile.config.

\# CUDA architecture setting: going with all of them.
\# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
\# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
                -gencode arch=compute_20,code=sm_21 \
                -gencode arch=compute_30,code=sm_30 \
                -gencode arch=compute_35,code=sm_35 \
                -gencode arch=compute_50,code=sm_50 \
                -gencode arch=compute_52,code=sm_52
\#               -gencode arch=compute_60,code=sm_60 \
\#               -gencode arch=compute_61,code=sm_61 \
\#               -gencode arch=compute_61,code=compute_61
prehensilecode commented 7 years ago

If using cmake, look for CUDA_ARCH_NAME. It can be set to "Manual" if your arch is not listed. After a Configure, CUDA_ARCH_BIN and CUDA_ARCH_PTX will be available.

Huxwell commented 7 years ago

For me the vanilla file didn't have compute_60 at all:

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
        -gencode arch=compute_20,code=sm_21 \
        -gencode arch=compute_30,code=sm_30 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_50,code=sm_50 \
        -gencode arch=compute_50,code=compute_50

And yet I get: nvcc fatal : Unsupported gpu architecture 'compute_60'

Anybody has any ideas what else can I try?

ycsun19 commented 7 years ago

I have the same problem with you, have you solved it yet? @Huxwell

Huxwell commented 7 years ago

I works only on machines with gtx1070 or gtx1080 for me. No chances for titan x, gtx 770, cpu for me - no matter what CUDA_ARCH is specified. I am still struggling to figure out the reason.

Ashutosh1995 commented 7 years ago

Even after commenting, while building, this error pops up on CUDA 7.0 with cuDNN 5.0. on Ubuntu 14.04 Is there any solution to this

Huxwell commented 7 years ago

It might concern only me, but if there is any chance that you are using some nvidia-docker dockerfile from different repo, please double check if you really use the Makefile.config file that you edited, not the default one which gets automatically renamed by a script or some file downloaded from github during the docker building process.

Ashutosh1995 commented 7 years ago

I downloaded the caffe version from the caffe github website and then changed the Makefile.config using CUDA 7.0 and cuDNN 5.0.

I followed the standard procedure of mv makefile.config.example makefile.config to make changes and then using build directory to build it.

silrep2 commented 7 years ago

I solve this problem by following: cd /usr/local/lib sudo rm cuda sudo ln -s cuda-8.0/ cuda

chriskraus0 commented 7 years ago

Hello Everyone,

I am using CUDA 9.0 on Ubuntu 16.04 and I am facing the following error:

nvcc fatal   : Unsupported gpu architecture 'compute_20'

This can be resolved by removing the lines in the Makefile.config :

-gencode arch=compute_20,code=sm_20 \
 -gencode arch=compute_20,code=sm_21 \

Best, Chris

jonatascbl commented 6 years ago

Where can i find the Makefile.config? I have the same problem, but i can't find this archive to edit.

r3lik commented 6 years ago

Can't find it either.

soulslicer commented 6 years ago

I mean..Caffe is sort of dead now. Everybody is publishing work to TF now

jonatascbl commented 6 years ago

In chriskraus0 say: This can be resolved by removing the lines in the Makefile.config :

-gencode arch=compute_20,code=sm_20 \ -gencode arch=compute_20,code=sm_21 \

But i can't find the Makefile.config to edit. How can i edit? I need make this archive. I don't understand, i am beginner.

chriskraus0 commented 6 years ago

Hello,

the file is in the base folder of caffe, or you have to create it by copying/renaming Makefile.config.example.

Have a look at the installation instructions ( http://caffe.berkeleyvision.org/installation.html#compilation).

Best wishes, Chris

jonatascbl commented 6 years ago

Thank you so much!

Saturday32 commented 6 years ago

using cmake to build caffe, CUDA 8, cuDNN V7, 1070 Ti when i use cmake it shows cuda 7.5 even though i have cuda 8.0 installed

"[ 1%] Built target caffeproto [ 2%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_prelu_layer.cu.o nvcc fatal : Unsupported gpu architecture 'compute_61' CMake Error at cuda_compile_generated_prelu_layer.cu.o.cmake:207 (message): Error generating /home/stanley/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_prelu_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:483: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_prelu_layer.cu.o' failed make[2]: [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_prelu_layer.cu.o] Error 1 CMakeFiles/Makefile2:304: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed make[1]: [src/caffe/CMakeFiles/caffe.dir/all] Error 2 Makefile:127: recipe for target 'all' failed make: *** [all] Error 2 "

please help

javierselva commented 6 years ago

I am having a similar problem. I am trying to install Caffe into a Docker image. I am doing so for trying out some code for Gaze Analysis. I am using Cuda 7.5 and CuDNN 5 on Ubunut 14. I have modified the GPU Dokerfile in this repository to use those. As suggested I have removed these lines from Makefile.config:

-gencode arch=compute_60,code=sm_60 \
-gencode arch=compute_61,code=sm_61 \
-gencode arch=compute_61,code=compute_61

I've also made sure that the version of NCCV I am using is compatible with the Cuda and CuDNN versions. However, when the process gets to the point of running make -j"$(nproc)" I still get:

-- Build files have been written to: /opt/caffe/build
[  1%] Running C++/Python protocol buffer compiler on /opt/caffe/src/caffe/proto/caffe.proto
Scanning dependencies of target proto
[  1%] Building CXX object src/caffe/CMakeFiles/proto.dir/__/__/include/caffe/proto/caffe.pb.cc.o
Linking CXX static library ../../lib/libproto.a
[  1%] Built target proto
[  2%] [  2%] [  2%] [  2%] [  2%] [  4%] [  4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_math_functions.cu.o
Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_adagrad_solver.cu.o
Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_sgd_solver.cu.o
Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_rmsprop_solver.cu.o
Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_adadelta_solver.cu.o
Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_nesterov_solver.cu.o
[  4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_adam_solver.cu.o
Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_contrastive_loss_layer.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_60'
nvcc fatal   : Unsupported gpu architecture 'compute_60'
CMake Error at cuda_compile_generated_adadelta_solver.cu.o.cmake:206 (message):
  Error generating
  /opt/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/solvers/./cuda_compile_generated_adadelta_solver.cu.o

Which is reasonable as cmake -DUSE_CUDNN=1 -DUSE_NCCL=1 .. outputs this at some point:

-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_20 sm_21 sm_30 sm_35 sm_50 sm_60 sm_61
--   cuDNN             :   Yes (ver. 5.1.10)

Which makes me think that it is completely ignoring Makefile.config as sm_52 does not appear here while it does in said file. However, something very weird happens if I try to do this from within a Docker container. So I built an image that performed all steps in the Dockerfile up to creating the build folder. Then I ran that image with nvidia-docker run --rm -it --entrypoint /bin/bash caffe_image and manually executed these commands:

cd /opt/caffe/
mkdir build
cd build
cmake -DUSE_CUDNN=1 -DUSE_NCCL=1 ..

In this setting, the output is this:

-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_50
--   cuDNN             :   Yes (ver. 5.1.10)

As the architecture is supported this finishes without problems and so does make -j"$(nproc)". However I am still unable to compile this properly for the rest of GPU architectures.

Anyone has any idea on what is going on? Or how could this be solved? For reference I've left the Dockerfile I am using and the modified version of the Makefile.config here.

Thanks in advance.

naibaf7 commented 6 years ago

@javierselva It looks like you are not compiling with the Makefile, but rather you are using CMake. In that case, this is a different build system and you need to remove/change the GPU archs with something like cmake-gui. Cmake does NOT read the Makefile.config, which is only used if you execute "make" in the root directory of Caffe (different build system).

javierselva commented 6 years ago

Great! Thank you very much, @naibaf7, I am a bit new to compiling stuff and I still get confused with the differences between Make and CMake. I'll research it for future issues.

So following your response, I modified the cmake/Cuda.cmake file and removed the architectures 60 and 61 in lines 7 and 92:93. Now everything seems to be working perfectly.

Thanks again!!

naibaf7 commented 6 years ago

@Noiredd I think it's time to abandon the Makefile build system, due to issues like @javierselva is reporting. What do you think?

Noiredd commented 6 years ago

@naibaf7 I don't have a clear opinion on that, mostly due to my limited experience with make/cmake. Still, having a single build system would likely limit the failure points, so it might be a good thing to do.

naibaf7 commented 6 years ago

@Noiredd At least for the OpenCL branch, the Makefile infrastructure has become unmaintainable and it will be removed there soon. Also people get easily confused between the two, as evident here. Makefiles are just not powerful enough to pick up all the dependency paths, it often involves a lot of manual work.

yonglianglan commented 6 years ago

For windows ?

GitEasonXu commented 6 years ago

@prehensilecode I don't know very well about your explanation: CUDA_ARCH_NAME. It can be set to "Manual" How can I set CUDA_ARCH to arch=compute_50,code=sm_50.

Look forward to your reply.

sahlex commented 1 month ago

I had the same problem when compiling torchaudio. I solved it by setting export TORCH_CUDA_ARCH_LIST=8.6 to an explicit version.

Hope that helps!

dragonQian commented 1 month ago

这是来自QQ邮箱的假期自动回复邮件。   您好,我最近正在休假中,无法亲自回复您的邮件。我将在假期结束后,尽快给您回复。