PyCaffe backward() produces zero gradients when GPU mode

[fferrara]

I'm working on generating saliency maps though gradient data, as in this famous paper.

It works well when using pycaffe in CPU mode. When in GPU mode, all the gradients computed by the backward pass are zeros.

I'm using Ubuntu 14.04, VGG Net without a Data Layer and I set force_backward : true.

This is the code that works

import caffe
import numpy as np
caffe.set_mode_gpu()
net = caffe.Net('deploy_saliency.prototxt', 'models/snapshot.caffemodel', caffe.TEST)

in_ = 'data'
transformer = caffe.io.Transformer(
                {in_: net.blobs[in_].data.shape})
transformer.set_transpose(in_, (2, 0, 1))
transformer.set_mean(in_, np.array([104, 117, 123]))
transformer.set_raw_scale(in_, 255)
transformer.set_channel_swap(in_, (2, 1, 0))
img = 'example_class_2.jpg'
image = caffe.io.load_image(img)
input_ = transformer.preprocess(in_, image)
net.blobs[in_].data[...] = input_
net.forward()
label = np.zeros((1, 5))
label[0,2] = 1
net.backward(**{'prob':label})

This works, indeed

import caffe
import numpy as np
caffe.set_mode_cpu()
net = caffe.Net('deploy_saliency.prototxt', 'models/snapshot.caffemodel', caffe.TEST)

in_ = 'data'
transformer = caffe.io.Transformer(
                {in_: net.blobs[in_].data.shape})
transformer.set_transpose(in_, (2, 0, 1))
transformer.set_mean(in_, np.array([104, 117, 123]))
transformer.set_raw_scale(in_, 255)
transformer.set_channel_swap(in_, (2, 1, 0))
img = 'example_class_2.jpg'
image = caffe.io.load_image(img)
input_ = transformer.preprocess(in_, image)
net.blobs[in_].data[...] = input_
net.forward()
label = np.zeros((1, 5))
label[0,2] = 1
net.backward(**{'prob':label})

[longjon]

Closing as there is not enough information here to suggest a bug; how are you reading the gradient?

If there's still a problem you can reproduce with the latest Caffe master, feel free to open a new issue with instructions to recreate it; see https://github.com/BVLC/caffe/blob/master/CONTRIBUTING.md.

[ambarpal]

@fferrara Were you able to resolve it? I'm also facing the same error

[fferrara]

Actually, I kind of gave up. I didn't try the latest master branch, because I can still use CPU for producing saliency maps.

[ambarpal]

I see. I also did give up earlier, but now have to look for a way to fix it since the CPU is getting too slow for me. I'll try this with the master branch and create a new issue there.