Hopfield Network [No hidden layer]

[JellePiepenbrock]

Is it possible to use Paysage to make a classical Hopfield network, without hidden units? In the examples, there is a Hopfield Network for MNIST, but this has hidden units. In trying to remove it (either setting n_hidden to 0 or removing the hidden layer from the layer list), there are a number of errors, which could be either mostly related to the fact that the utils/plotting files in the examples assume there is a hidden layer.

[drckf]

Every model in Paysage has at least one hidden layer.

As a result, there are two things to consider with regards to the Hopfield model.

1. The equilibrium distribution obtained using a Bernoulli visible layer and a Gaussian hidden layer is the same as a classical Hopfield network.

This is easy to show by integrating out the hidden units to compute the energy of the visible units:

E(v) = - log p(v) 
     = - log(\int dh p(v, h))
     = C - \log(\int dh e^( \sum_i a_i v_i + \sum_{\mu} h_{\mu}^2/2 + \sum_{i \mu} W_{i \mu} v_i h_{\mu}) 
     = C - \sum_i a_i v_i + \sum_{\mu} \log( \int dh_{\mu} e^{ h_{\mu}^2/2 + \sum_i W_{i \mu} v_i h_{\mu})
     = C' - \sum_i a_i v_i - 1/2 * \sum_{ij} (\sum_{\mu} W_{i \mu} W_{j \mu} ) v_i v_j

The constant term isn't relevant for the probability, so this is the same as the classical Hopfield model.

2. If we consider the dynamics of a classical Hopfield model to be single spin flips (i.e., Glauber updates), then the dynamical properties of the Bernoulli-Gaussian RBM (which is updated by block Gibbs sampling) will be very different.

TLDR: you can use Paysage to compute properties of the equilibrium distribution of a classical Hopfield model but you cannot use it to compute dynamical properties with single spin flips.