When I run net on other gpu which is not 0, an CuDNNConvolutionLayer Allocate error will happen

[HaoLiuHust]

I want to use nvcaffe in gRPC, but when call net.Forward, the program will faill in caffe::GPUMemory::Workspace::safe_reserve, the call stack is:
caffe::GPUMemory::Workspace::safe_reserve(unsigned long, int) 0x00007fffe3814a5e caffe::CuDNNConvolutionLayer<float, float>::AllocateWorkspace(unsigned long) 0x00007fffe348a277 caffe::CuDNNConvolutionLayer<float, float>::Reshape(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&) 0x00007fffe348fde7 caffe::Layer<float, float>::Forward(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&) 0x00007fffe3388dda caffe::Net::ForwardFromTo(int, int) 0x00007fffe378428b caffe::Net::Forward(float*) 0x00007fffe3784426

[HaoLiuHust]

Any one can help?

[drnikolaev]

@HaoLiuHust do you have display connected?

[HaoLiuHust]

@drnikolaev I am connecting the server with vnc, and if I run the model on gpu 0, it will be ok. The error only occur when I want to run it on other gpus

[drnikolaev]

@HaoLiuHust could you please check this rc https://github.com/drnikolaev/caffe/tree/caffe-0.17

[HaoLiuHust]

@drnikolaev Thank you very much, I will try it

[drnikolaev]

Please verify https://github.com/NVIDIA/caffe/tree/v0.17.1