How to handle multi label multiclass?

[rrags]

My network does binary classifcation for detection of 11 different classes. E.G. It predicts if or if not there are apples, oranges, pineapple, and pears in the image I give it. So the output is a binary vector of length 11.

Can I use this project with out modifying it? I have checked and top_label will always be 9, so what I put for target_label will be ignored. I get the error

The predicted class index 9 does notequal the target class index 3. Calculatingthe gradient w.r.t. the predicted class. 'the gradient w.r.t. the predicted class.'

So in my example, if the network outputs a binary vector of length 4 corresponding to whether or not apple, orange, pineapple, and pear are in the image, how can I make it so that when I set target = 3 the code will show the gradient corresponding to the task of detecting pineapples?

Also, I am using modified ResNet-18 (transfer learned).

[dnns92]

I think you can hack the script at backprob.py to generate the target vector you are looking for:

    def calculate_gradients(self,
                            input_,
                            target_class=None,
                            take_max=False,
                            guided=False,
                            use_gpu=False):

       (...some unrelated code...)

            target = torch.FloatTensor(1, output.shape[-1]).zero_()

      (...some unrelated code...)

            target[0][top_class] = 1

and change that to something like:

    def calculate_gradients(self,
                            input_,
                            target_vector=None, <--- your target vector goes here
                            take_max=False,
                            guided=False,
                            use_gpu=False):

          if isinstance(target_class, torch.tensor):
                  target = target_vector  # in this case, you have to generate the target-vector yourself. 
          else:
                 (continue as before)

Then you also have to change the respective parent functions that call this function such that you can pass your own target vector, but this should be easy.

In that way you can set a arbitrary target vector, which can also contain multiple "1"-entries. This might achieve what you are looking for. I am not deep enough in the math behind saliency maps though. i hope @MisaOgura can comment on this.

[MisaOgura]

Hi @rrags, apologies for the late reply and thanks @dnns92 for jumping in.

The current behaviour is that if target_class and top_class are different, it defaults to visualise the gradients w.r.t the top_class (see here). The simplest hack to achieve what you want to do would be to change backprop.py#L117 from target[0][top_class] = 1 to target[0][target_class] = 1.

Allowing users to have more control over which class to visualise, is something I have been wanting to revisit - please expect an update on this.

@dnns92 The modification suggested would make sense when target_vector contains only one value of 1 (i.e. one target class). When there is more than one target class, .backward() would calculate gradients w.r.t multiple target classes at once, and you will only get one set of gradients out of the graph. This probably wouldn't be the desired behaviour when creating per-class saliency map, so multiple targets would need to be handled differently. Hope this helps.

[rrags]

Okay thanks. This is pretty much what I was doing and just wanted to make sure I was on the right track and get your insight.