GaussianBinaryRBM gets very similar fantasy particles

[bhomass]

I created my own GaussianBinaryRBM

function GaussianBinaryRBM(sz::Union{Int,Dims}, szb::Union{Int,Dims})
    b = zeros(Float64, szb...)
    w = zeros(Float64, sz..., szb)
    return RBM(Gaussian(Float64, sz), Binary(; θ = b), w)
end

and fed real valued MNIST data into it. after training and generate fantasy particles by pcd, I get synthesized characters back, in addition to being noisy, there is no variability in the output images.

is there simple reason why this is?

[bhomass]

I played with the parameters a lot more. It appears the model never stabilizes with more training. It oscillates between high noise and higher variability, and low noise and almost no variability. This goes to the criticism I heard before, that a gaussian distribution may be just too simplistic to represent the input visible data.

[cossio]

I have not tried before this combination of layers. Something that has helped me in other settings is the centering trick (and for continuous units, to standardise their variance too). You can try https://github.com/cossio/StandardizedRestrictedBoltzmannMachines.jl. I'd be curious to know how it works for you.