frankseide
commented
8 years ago OK, it seems that we cannot do this presently. The reason is that RBM pre-training is no longer necessary. Let me analyze it, to see what one would have to add to enable CD-1.
One missing piece is the sampling. uniform_dist is only randomly initialized at startup, and never gets updated. This will require C++ code to be written. I am wondering whether we can wing it with Dropout(), but probably not.
The other missing piece is reduction over samples. To write the per-sample gradient (Wvh), you would say (translating your formula):
grad = TransposeTimes (features, pos_hid_prob) - TransposeTimes (features_reconstruction, neg_hid_prob)which would then have to be reduced over all samples in the minibatch. Unfortunately, we would first have to compute the above products for every sample, which will be very big in memory, and inefficient. Internally, we already have code that can multiply and reduce directly, but there is presently no way to tell on BS level.
As for the criterion, below I copy the gradient code and source-code comment of the DummyCriterionNode below. You would need to supply 3 parameters:
- the objective values--these are only reported in the log, but don't affect the gradient
- the gradient--your formula must compute the gradient for the parameters
- the parameter you want to update (
Wvh,Bv, orBh)
So my apologies that I cannot offer a better answer at this point in time. I will raise this interally and see if and how we can address the above two gaps in a future version.
From the source code of DummyCriterionNode:
// predictionsGradient += userSuppliedGradient * scalarGradientFromTop
auto gradient = Input(2)->GradientFor(fr);
Matrix<ElemType>::Multiply1x1AndWeightedAdd(+1.0f, /*gradient from top:*/Gradient() /*1x1*/,
/*user-supplied gradient:*/Input(1)->ValueFor(fr), 1.0f,
/*add to:*/gradient);
// DummyCriterionNode (objectiveValues, userSuppliedGradient, prediction)
//
// Apply user-supplied gradient, computed as Forward(), as the gradient into 'prediction'.
//
// predictionsGradient += userSuppliedGradient * scalarGradientFromTop
//
// This training criterion node allows to compute objectives and gradient
// with custom CNTK expressions (as Forward() computations). It has 3 inputs:
// 1. custom objective values to be summed up and passed up
// 2. custom gradient values to be passed down as the gradient into 'prediction'
// 3. prediction: the node to pass the custom gradient into
zpbappi
I am trying to write an RBM with CD1 learning. It does have an error function defined by
Sum(features - v1_reconstruction).^2). However, my goal is _NOT_ to adapt the weights so that the error function is minimized. Rather, I would like to follow a specific update rule to update weights and biases. FYI, for CD1, there will be 3 parameters that I need to learn: weight between visible and hidden units and two biases in both directions.I was hoping to use DummyCriterion. Looking at the source code, it expects _objectivefunction, gradient and _target_predictionnode. I thought this node will update "something" with the supplied gradient (of course, applying proper momentum and learning rate from SGD parameters). I am kind of stuck here. I am not entirely sure how I tell it to update the weights parameter with the gradient I supply? There may be more than one such parameter in a network and they all may have different gradient calculation.
Here is my network expression in (sudo)BS:
Now, the weight update should be a function of
features,feature_reconstruction,pos_hid_probandneg_hid_prob. I think, the gradient should be average offeatures' * pos_hid_prob - features_reconstruction' * neg_hid_prob. And, similar equations may be written for both visible bias unit (Bv) and hidden bias unit (Bh). As you can see, if I want to declare the gradients when declaringWvh,BvandBh, I end up in a situation with circular dependency. Or, am I missing something and doing it the wrong way?I do not quite understand how to proceed further to implement an RBM. I would really appreciate any help/direction.