search

kyunghyuncho / deepmat

Matlab Code for Restricted/Deep Boltzmann Machines and Autoencoders
http://users.ics.aalto.fi/kcho/
188 stars 95 forks source link

Gaussian - Bernoulli RBM #7

Closed subhaprakash closed 10 years ago

subhaprakash commented 10 years ago

Hi,

  I am following your master thesis to understand RBM and its variants. I am trying to implement krizhevsky's  Gaussian -bernoulli RBM in matlab. When i do, as you said in your document i am getting sigmas in -ve range. After 10 batch, i am getting error as inf .When i visualize weights , data and reconstructed data, resconstructed data appears like white or black pixels. I could not see any image.

my sigma learning rate=10^-8. U might have come across this kind of issu. Could u able to suggest me?

kyunghyuncho commented 10 years ago

Hi,

As I found that learning sigmas is extremely sensitive to the choice of learning rate and its scheduling, I almost always used adaptive learning rate to train a Gaussian-Bernoulli RBM.

Does it happen also with the adaptive learning rate? If so, I'd suggest you to set the lower- and upper-bounds of sigmas to some conservative values (0.1 and 1.1).

Also, Often, it helped me to whiten the data or simply make each pixel/dimension of data to be zero-mean and unit-variance.

Best,

On Tue, Aug 19, 2014 at 9:24 PM, subhaprakash notifications@github.com wrote:

Hi,

I am following your master thesis to understand RBM and its variants. I am trying to implement krizhevsky's Gaussian -bernoulli RBM in matlab. When i do, as you said in your document i am getting sigmas in -ve range. After 10 batch, i am getting error as inf .When i visualize weights , data and reconstructed data, resconstructed data appears like white or black pixels. I could not see any image.

my sigma learning rate=10^-8. U might have come across this kind of issu. Could u able to suggest me?

— Reply to this email directly or view it on GitHub https://github.com/kyunghyuncho/deepmat/issues/7.

subhaprakash commented 10 years ago

Thanks for your immediate response .I have not used adaptive learning rate yet. As i just started learning Gaussian RBM concepts, I implemented simple version(following hinton et al's code) of krizh Gaussian Bernoulli RBM using CD1. I have a problem in representing p(v|h)= bi+sigma(hj * wij). Initially i am trying to implement w/o any learning update for sigma and simply assigned fixed value as 1 and even tried with sigma update, i get same problem. if you don't mind can i send you that piece of code.. I am not able to discuss with anyone. I feel these below four lines have some problem. I am breaking my head for long.

This is the code i used for negative phase.

poshidstates= poshidprobs > rand(numcases,numhid); negdatapart= poshidstates_vishid'; %bsxfun(@rdivide,,sigmas)); negdatapart2= negdatapart_repmat(sigmas, numdims,1)'; negdata= bsxfun(@plus,negdatapart2,repmat(visbiases,numcases,1))

thanks for your time

Subha Manoharan

On Thu, Aug 21, 2014 at 2:45 AM, Kyunghyun Cho notifications@github.com wrote:

Hi,

As I found that learning sigmas is extremely sensitive to the choice of learning rate and its scheduling, I almost always used adaptive learning rate to train a Gaussian-Bernoulli RBM.

Does it happen also with the adaptive learning rate? If so, I'd suggest you to set the lower- and upper-bounds of sigmas to some conservative values (0.1 and 1.1).

Also, Often, it helped me to whiten the data or simply make each pixel/dimension of data to be zero-mean and unit-variance.

Best,

  • Cho

On Tue, Aug 19, 2014 at 9:24 PM, subhaprakash notifications@github.com wrote:

Hi,

I am following your master thesis to understand RBM and its variants. I am trying to implement krizhevsky's Gaussian -bernoulli RBM in matlab. When i do, as you said in your document i am getting sigmas in -ve range. After 10 batch, i am getting error as inf .When i visualize weights , data and reconstructed data, resconstructed data appears like white or black pixels. I could not see any image.

my sigma learning rate=10^-8. U might have come across this kind of issu. Could u able to suggest me?

— Reply to this email directly or view it on GitHub https://github.com/kyunghyuncho/deepmat/issues/7.

— Reply to this email directly or view it on GitHub https://github.com/kyunghyuncho/deepmat/issues/7#issuecomment-52788161.

subhaprakash commented 10 years ago

Hi,

  Sorry cho, i found the mistake just after posted you. Thanks for your

time.. Sorry to disturb you. When i changed above code like this , it works out fine. Because,i need to elementwise multiplication. It may be helpful for some one else..

negdatapart2= bsxfun(@times,sigmas, negdatapart)

thanks Subha Manoharan

On Thu, Aug 21, 2014 at 4:03 PM, subha krishna subhakrishh@gmail.com wrote:

Thanks for your immediate response .I have not used adaptive learning rate yet. As i just started learning Gaussian RBM concepts, I implemented simple version(following hinton et al's code) of krizh Gaussian Bernoulli RBM using CD1. I have a problem in representing p(v|h)= bi+sigma(hj * wij). Initially i am trying to implement w/o any learning update for sigma and simply assigned fixed value as 1 and even tried with sigma update, i get same problem. if you don't mind can i send you that piece of code.. I am not able to discuss with anyone. I feel these below four lines have some problem. I am breaking my head for long.

This is the code i used for negative phase.

poshidstates= poshidprobs > rand(numcases,numhid); negdatapart= poshidstates_vishid'; %bsxfun(@rdivide,,sigmas)); negdatapart2= negdatapart_repmat(sigmas, numdims,1)'; negdata= bsxfun(@plus,negdatapart2,repmat(visbiases,numcases,1))

thanks for your time

Subha Manoharan

On Thu, Aug 21, 2014 at 2:45 AM, Kyunghyun Cho notifications@github.com wrote:

Hi,

As I found that learning sigmas is extremely sensitive to the choice of learning rate and its scheduling, I almost always used adaptive learning rate to train a Gaussian-Bernoulli RBM.

Does it happen also with the adaptive learning rate? If so, I'd suggest you to set the lower- and upper-bounds of sigmas to some conservative values (0.1 and 1.1).

Also, Often, it helped me to whiten the data or simply make each pixel/dimension of data to be zero-mean and unit-variance.

Best,

  • Cho

On Tue, Aug 19, 2014 at 9:24 PM, subhaprakash notifications@github.com wrote:

Hi,

I am following your master thesis to understand RBM and its variants. I am trying to implement krizhevsky's Gaussian -bernoulli RBM in matlab. When i do, as you said in your document i am getting sigmas in -ve range. After 10 batch, i am getting error as inf .When i visualize weights , data and reconstructed data, resconstructed data appears like white or black pixels. I could not see any image.

my sigma learning rate=10^-8. U might have come across this kind of issu. Could u able to suggest me?

— Reply to this email directly or view it on GitHub https://github.com/kyunghyuncho/deepmat/issues/7.

— Reply to this email directly or view it on GitHub https://github.com/kyunghyuncho/deepmat/issues/7#issuecomment-52788161.