Dagnn Customized loss Layer

[pramodyn]

How can i define a Customized Loss Layer in DAGnn setup? In a Simplenn framework, it was easy by defining the Forward and Backward functions. How do we do it in DAGnn Framework?

[luzhi]

I have the same question ...

[Nicwhu]

Hi, I met the same question. Have you found any solutions? Thanks a lot if you can tell me!

[AruniRC]

Here are some code examples. I needed to have a regularisation term that would penalise the L1 norm of activations. It made sense to implement this as a Loss function.

First, because of DAG, I needed to create a subclass of Loss:

classdef LossRegul < dagnn.Loss
%LOSSREGUL Loss function for regularisation (L1) of features X
%   Detailed explanation goes here

properties
    type = 'L1'
end

methods
    function outputs = forward(obj, inputs, params)
      outputs{1} = vl_nnlossregul(inputs{1}, [], obj.type) ;
      n = obj.numAveraged ;
      m = n + size(inputs{1},4) ;
      obj.average = (n * obj.average + gather(outputs{1})) / m ;
      obj.numAveraged = m ;
    end

    function [derInputs, derParams] = backward(obj, inputs, params, derOutputs)
      derInputs{1} = vl_nnlossregul(inputs{1}, derOutputs{1}, obj.type) ;
      derParams = {} ;
    end

    function outputSizes = getOutputSizes(obj, inputSizes, paramSizes)
      outputSizes{1} = [1 1 1 inputSizes{1}(4)] ;
    end

    function rfs = getReceptiveFields(obj)
      % the receptive field depends on the dimension of the variables
      % which is not known until the network is run
      rfs(1,1).size = [NaN NaN] ;
      rfs(1,1).stride = [NaN NaN] ;
      rfs(1,1).offset = [NaN NaN] ;
      rfs(2,1) = rfs(1,1) ;
    end

    function obj = LossRegul(varargin)
      obj.load(varargin) ;
    end
end
end

The actual logic of calculating the forward and backward pass of the loss went into a vl_nn..... file, same as we did in SimpleNN:

function y = vl_nnlossregul(x, dzdy, type)
%VL_NNLOSSREGUL Calculate the loss from regularising a feature X
%   Only supports L1 (LASSO) regulariser for now

% TODO - different types: L1 and L2 regularizations

backMode = ~isempty(dzdy);

if backMode
    % derivatives
    y = bsxfun(@times, sign(x), dzdy) ; 
else
    % sum the loss over spatial and feature (channel) dimensions
    %   -- also sums over batches
    y = sum(sum(sum(sum(abs(x), 1), 2), 3),4); % L1
end

Then I would add this to a network in the same way that a SoftMaxLoss() would be added. This code snippet is heavily based off the way MatConvNet itself writes the subclasses of Loss.

Hope this helps!

[MaxChu719]

In your example, you just have the x as input to the loss function. Could you also give another example that accept x(output of the net, may/may not be softmax) and c (ground truth label) in normal cases.

[AruniRC]

Here's the class definition for L2 loss

classdef L2Loss < dagnn.Loss
%LOSSREGUL Loss function for L2 loss between targets and activations
%   Detailed explanation goes here

properties
    type = 'L2'
end

methods
    function outputs = forward(obj, inputs, params)
      outputs{1} = vl_nnL2(inputs{1}, inputs{2}, []) ;
      n = obj.numAveraged ;
      m = n + size(inputs{1},4) ;
      obj.average = (n * obj.average + gather(outputs{1})) / m ;
      obj.numAveraged = m ;
    end

    function [derInputs, derParams] = backward(obj, inputs, params, derOutputs)
      derInputs{1} = vl_nnL2(inputs{1}, inputs{2}, derOutputs{1}) ;
      derInputs{2} = [] ;
      derParams = {} ;
    end

    function outputSizes = getOutputSizes(obj, inputSizes, paramSizes)
      outputSizes{1} = [1 1 1 inputSizes{1}(4)] ;
    end

    function rfs = getReceptiveFields(obj)
      % the receptive field depends on the dimension of the variables
      % which is not known until the network is run
      rfs(1,1).size = [NaN NaN] ;
      rfs(1,1).stride = [NaN NaN] ;
      rfs(1,1).offset = [NaN NaN] ;
      rfs(2,1) = rfs(1,1) ;
    end

The vl_nnL2() is based off #15

[yangcong955]

Hi @AruniRC could you please tell us how to add L2Loss into the DagNN net? Thank you very much

[AruniRC]

@yangcong955 hope this helps:

At the end of your network, let us suppose that the final output variable of your network is called y. Your read in the target or ground-truth vectors as label (this will have to be handled in the getBatch...() function).

net.addLayer('l2_loss', L2Loss(), {'y', 'label'}, {'objective'});

The layer is named l2_loss, it is defined by the object L2Loss(), the inputs are y and label. The output of your loss layer is called objective. This is handy because by default MatConvNet treats the variable objective as the loss during backprop by default.

In short adding a new layer involves:

• [ ] create a vl_nnL2() layer that handles the actual forward and backward calculations

• [ ] create a class that wraps around the vl_nnL2() function, a la the original MatConvNet dann.DagNN.Loss().

• [ ] Add the new layer to your network using net.addLayer(...). Take care on how you name your variables. Having net.layers, net.vars and net.params open in your Matlab GUI makes life much easier for this. :)

• [ ] Figure out what your inputs to the network should be depending on your loss layer. Modify your getBatch..() function accordingly to provide the correct inputs to the network with the correct naming.

N.B. -- a good debugging technique would be to set to precious all the params of the network right after you have finished defining it, and then make sure it runs on a single input (image, label) pair. Look at the net.vars to check dimensionality and generaly lack of messiness.

[yangcong955]

Hi @AruniRC Thank you so much for your detailed reply. Really appreciate!

[yangcong955]

Hi @AruniRC , is that possible for you to provide an example getBatch function for DagNN eval()? I use exactly the same (name, setup, etc) loss function as you provided. The net input is 'data'. Batchsize is 300. My imdb.images is organised by data (224x224x3x10000), label (1x1x7x10000), id(1x10000), set(1x10000). Thank you very much!

[AruniRC]

I don't have exactly the same setup as you so I can't share my getBatch code.

Take a look at your net.layers. What is/are the input variables? Let's assume your network takes in the images in a variable named 'input' (this is the default). So after a bunch of layers, your network's output is 1x1x7. Let's call this 'y'. Suppose your network loss takes 'y' and 'label' and computes their Euclidean distance. The output of your loss layer is named 'objective'.

You can verify that this is indeed the case by looking at net.layers, net.vars etc.

Your getBatch() should return the inputs to your network. Take a look at the getBatchDagNN..() in the examples.

Your getBatch() should return a cell like this:

`inputs = {'input', data, 'label', label};

Now eval this in your network. Hope this helps.

[yangcong955]

@AruniRC It works now, thank you very much!

[h612]

Excellent response. I have a question regarding regression and getBatch. Im performing counting using regression. The label includes image counts. Once i calculate the l2 loss how can i change the getBatch function?

[jessicaloohw]

Hi @AruniRC,

I am writing a custom loss function for MatConvNet using the DagNN wrapper as well i.e. I'm replacing the vl_nnloss with a custom_loss function. I'm a bit confused as to what the function should return during the backward pass i.e. 'DZDX = VL_NNLOSS(X, C, DZDY)`.

Do I just calculate DYDX as the derivative of my custom loss function w.r.t. its input and multiply by DZDY?

Thanks!