search

skaae / rbm_toolbox_lua

Port of RBM_toolbox to lua
BSD 3-Clause "New" or "Revised" License
38 stars 20 forks source link

Initialisation of W and U from a uniform distribution is incorrect #8

Open elezar opened 8 years ago

elezar commented 8 years ago

In the paper "Learning Algorithms for the Classification Restricted Boltzmann Machine", a uniform distribution in the interval [-m**-0.5, m**-0.5] is used for the initialisation of the weight matrices (rbm.W and rbm.U).

However, in the initcrbm the following code is used:

M = math.max(m,n);
interval_max = math.pow(M,-0.5);
interval_min = -interval_max;
weights = torch.rand(m,n):mul( interval_min + (interval_max-interval_min) )

which initialises the weights using the interval [0, m**-0.5].

Should the calculation of the weights not be?

weights = torch.rand(m,n):mul(interval_max-interval_min):add(interval_min)
skaae commented 8 years ago

Thats true. Cant you just use uniform(-m^(-0.5), m^(-0.5)) ?

elezar commented 8 years ago

I'm new to lua, so maybe the uniform function will give the right result. I will check.

On Tue, 17 Nov 2015 19:03 Søren Kaae Sønderby notifications@github.com wrote:

Thats true. Cant you jus use uniform(-m^(-0.5), m^(-0.5))

— Reply to this email directly or view it on GitHub https://github.com/skaae/rbm_toolbox_lua/issues/8#issuecomment-157454655 .