search

drckf / paysage

Unsupervised learning and generative models in python/pytorch.
Other
119 stars 25 forks source link

Deep models - WIP #79

Closed jrwalsh1 closed 7 years ago

jrwalsh1 commented 7 years ago

WIP, don't merge!

PR enables deep models to be trained. Adds a 3-layer RBM example for MNIST.

drckf commented 7 years ago

Here are a couple of suggestions for things to try:

I've noticed that people often use the mean_field_iteration for the positive phase even when using MCMC for the negative phase. For example, check out "Gaussian-Bernoulli Deep Boltzmann Machine" by KyungHyun Cho, Tapani Raiko and Alexander Ilin Section 3.1. Along the same lines, we may not want to use the DrivenSequentialMC sampler for the positive phase.

I think the idea is to just sample one mode of P(h | v) -- which could act like a kind of regularization by reducing the variance (at the expense of increased bias).