Classififer

[AurelienDecelle]

I was wondering about the necessity to put a classifier for the output of the RBM ? We can stack many RBM, but I (guess?) we cannot stack a classifier to test the predicting ability of our DBM. Should we put it ourself or use another library (I have seen that Mocha can do it but it looks a bit more complicated). In the former case, what would be the most convenient way to implement it, knowing that we will have to fine tune the network ?

A.

[eric-tramel]

@AurelienDecelle What are you thinking about for the classification method? So, we take a trained DBM, then, take some test set which we wish to classify, pass it through the DBM until we get to the highest level of hidden units, and then perform a classification on the activations of these hidden units, right?

Would you want to do this with a NN/DNN and backprop or do something simpler like a logistic regression? If you are thinking of using an NN, then we should probably just leave that up to other tools. I think, too, there must be some things out there already to perform the regression, as well.

I don't think we need to re-invent the wheel, one just needs to find the right classification package.

Or, are you thinking specifically of doing the DBN thing and unrolling the DBM and then performing backprop?

[AurelienDecelle]

@eric-tramel well, I was thinking about a naive logistic regression, but I'm afraid that the problem is slightly more involved than just "take a trained RBM and pass the highest level of hidden units to a classifier". From what I remember, to get good performance you have to add a "classifier" layer at the end of the RBM/DBM, and at some point fine-tune the whole network using backprop (still, if I remember well, if you train separatelly the RBM/DBM and the classifier you do not get very good performance, you really have to put the whole thing together in order to "specialize" the network).

The whole point being : I do not particularly want to redo what have already been done but if (if?) we want to test our RBM/DBM on some classification task we will need to fine-tune the network (so I guess doing backprop with the whole network).

A.

[krzakala]

@aurelien. This is exactly what mocha will do for you

F

— Envoyé avec Mailbox

On Fri, Sep 11, 2015 at 6:14 PM, AurelienDecelle notifications@github.com wrote:

@eric-tramel well, I was thinking about a naive logistic regression, but I'm afraid that the problem is slightly more involved than just "take a trained RBM and pass the samples to a classifier". From what I remember, to get good performance you have to add a "classifier" layer at the end of the RBM/DBM, and at some point fine-tune the whole network using backprop (still, if I remember well, if you train separatelly the RBM/DBM and the classifier you do not get very good performance, you really have to put the whole thing together in order to "specialize" the network). The whole point being : I do not particularly want to redo what have already been done but if (if?) we want to test our RBM/DBM on some classification task we will need to fine-tune the network (so I guess doing backprop with the whole network).

A.

Reply to this email directly or view it on GitHub: https://github.com/sphinxteam/Boltzmann.jl/issues/4#issuecomment-139588735

[AurelienDecelle]

Alright, so the answer is : we use Mocha :)

A.

[dfdx]

Note, that it's possible to export trained DBN / DBM to Mocha.jl for fine-tuning. If it makes sense, we can also create a function for automatic conversion to Mocha's network, e.g. something like this:

rbm = fit(GRBM(), X)
net = to_mocha(rbm)
fit(net, X, y)

Though it will create dependency on Mocha.jl, which makes sense only if somebody is actually going to use it.