VNNikolaidis
commented
1 year ago Thanks for the comments. There are several ways to do this, varying in code style (and runtime performance). After this reply I will have another with further alternative ways. The reply (a) that follows, is based on your code.
(a) So one way to compute softmax is by closely following your approach, i.e. use a customized layer. BTW your code is fine, it is 99.9% of the answer below. And I understand that it is hard to modify code that already exists.
I should mention here that it is easier to do customizing on a ‘generic’ layer. In nnlib2 there is a layer class called ‘pe_layer’, a generic layer. It is what is created when we call add_layer from R with ‘generic’ as layer type. It is described it as “a layer of generic "dumb" pes where most processing will be done in layer code”. I suspect this is what you want to use as base for your new layer.
But if BP functionality needs to be maintained then you have to modify a bp_comput_layer, i.e. a “BP-hidden” layer. This (in its current implementation) also consists of “dumb” PEs, i.e. most processing is done in layer code and not in PE (node) code. The following code does this:
class bp_comput_softmax_layer: public bp::bp_comput_layer
{
public:
void recall()
{
if(no_error())
{
DATA denom=0;
for(int i=0;i<size();i++)
{
pe REF p = pes[i];
DATA x = p.input; // input is already summated;
x = x + p.bias; // add bias,
denom=denom+exp(x);
}
for(int i=0;i<size();i++)
{
pe REF p = pes[i];
DATA x = p.input; // input is already summated;
x = x + p.bias; // add bias,
p.output=(DATA)1*exp(x)/denom;//
p.input=0; // reset input.
}
}
}
};All that was changed from your code is the class definition details. Then a name was added in generate_custom_layer function in file "additional_parts.h" (the name “BP-hidden-softmax” was chosen for the above). This was done as described in the documentation (as well as Step 3 here.
As this code has now been added to the additional_parts.h file found in Github, you can see the code there or use it if you install the package from there. Again, the name for creating a such layer when using the NN R module's add_layermethod is “BP-hidden-softmax”.
Some tests using this:
a <- new( "NN" ) # create a NN object in variable a
a$add_layer( "BP-hidden-softmax", 4 )
a$set_biases_at(1,c(0,0,0,0))
a$input_at(1,c(1,2,0.5,-2))
a$recall_at(1)
a$get_output_from(1)which returns:
> a$get_output_from(1)
[1] 0.22859235 0.62137844 0.13864827 0.01138094or
a$input_at(1,c(7,1,2,1))
a$recall_at(1)
a$get_output_from(1)which returns:
> a$get_output_from(1)
[1] 0.988439751 0.002450097 0.006660055 0.002450097Note that this based on the bp_comput_layer, it is meant to be a BP hidden layer. BP output layers are the same except they add their own encoding method. So a “bp-output-softmax” was also created (based on your code there as well). This was also added to “additional_parts.h”, but has not been tested much. Please let me know if you use it and have issues with it.
As you mentioned that you work on real data, if you plan to make the results public a reference to the package would be appreciated (For information on citing this package use the following R command: citation("nnlib2Rcpp") )
Another comment will follow this, with alternative suggestions you (or others) may want to consider.
tpq
Hi again, and thanks once more!
I've become more familiar with the package, and have started to use it to solve a real problem! I am training a multi-layer perceptron to predict multiple class labels (e.g., akin to a multinomial regression).
I am stuck now wanting to create an 'additional part', but I am unable to see how I can modify any of the existing templates from
additional_parts.h(whose examples focus on pes and connections).The part I desire is something that will perform a softmax transform across all pes within a layer. Reading over the documentation and codebase, I think this could be implemented as a custom layer, such as one modelled after the BP layer. If implemented, I imagine I could use the "BP-softmax" layer like:
where the final layer is a softmax transform of the previous output. I think this would have to be a distinct layer, and not a PE activation function, since the softmax calculation would need to be computed over a set of pes, not for a single pe.
I am raising an issue because I am a bit stuck on how best to approach creating a custom layer like this. Looking over the
nn_bp.cppfile, I thought I could create a new class layer likebp_comput_layer, that could softmax transform during recall. I suppose the operation might look something like (adapted frombp_comput_layer):I write to ask whether you have any ideas of a template I could follow as the easiest path toward implementing my own layers with custom encode and recall operations? (akin to the pes and connections in additional parts)? Sorry if this is obvious, I am a very weak C++ programmer, especially when it comes to object-oriented programming (but I am learning :D)!