Progress Anomaly Maps from RBM

[KJ-Waller]

So far, we've taken an RBM implementation from this repo. We trained an RBM on MNIST images of one class. Then, evaluating on another class, we make use of the Grumbel-softmax trick to convert the discrete latent variable to a continuous density function. This is then used to compute the score for backpropagation to visualize the attention maps as in the VAE case.

Some preliminary qualitative results, trained on digit 1, evaluated on digit 9:

[alfjesus3]

In order to improve the results we can follow the Geoff Hinton slides and add an extra hidden layer (see slide 4) - https://www.cs.toronto.edu/~hinton/csc321/notes/lec20.pdf Additionally with the extra layer we will be able to train longer

[alfjesus3]

Another thing we can make an ablation study is on the influence of the number of samples https://github.com/Carmoondedraak/FACT-2021/blob/8c0bc07a253bb5bf147033598b1b6717afeea385/rbm_anomaly_maps/RBM.py#L39 in the training. In this paper (table 1) varying to 20 yields higher results - https://arxiv.org/pdf/1805.07349.pdf

[KJ-Waller]

Below are some results from the 4 layered stacked RBM on MNIST, trained on digit 1 evaluated on digit 9.

The individual layer output dimensions are 484, 256, 64 and 16. It was trained for 80 epochs, initially with a single layer, adding a new layer every 20 epochs. We optimized using stochastic gradient descent, and a learning rate of 0.1 and batch size 64.

Visualizing from layer 0, with a Grumbel-softmax temperature of 0.6:

Visualizing from layer 1, with a Grumbel-softmax temperature of 0.6:

Visualizing from layer 2, with a Grumbel-softmax temperature of 0.6:

Visualizing from layer 3, with a Grumbel-softmax temperature of 0.6:

[alfjesus3]

On layer 3, there seems to start appearing some more precise patterns! The question now is if we should add more layers or just train for more epochs