search

sphinxteam / Boltzmann.jl

Restricted Boltzmann Machines in Julia
Other
15 stars 4 forks source link

Implement Weight Decay and Constraints #2

Closed eric-tramel closed 8 years ago

eric-tramel commented 8 years ago

The current version of Boltzmann.jl does not have an implementation of weight decay functionality, either in $\ell_1$ or $\ell_2$. We should include options for applying weight decay during the RBM training procedure.

The weight decay should be included within the function rbm.jl/update_weights!() as an extra gradient term when updating rbm.W. Additionally, an option for the weight decay strength should be included within the type definition of RBM.

Finally, options for specifying bounds on the weight magnitude should be included, as well. The implementation and option passing should be handled in the same way as the above.

eric-tramel commented 8 years ago

With Commit: 026febdd13bc72b4bf63255cffcd3aeee0c79a0c, we now fully support the use of both $\ell_1$ and $\ell_2$ weight decay/regularization. Progress!