Neural Net on Images

[Rohan2821999]

@cbattista, I ran the images data on a super simplified botzmann NN model (not a deep layer network) unsupervised. The acc is around 60%. I hence recreated the images from the output data.

Below is a comparison:

Actual Image Input:

Reproduced NN Image..

Looks to work pretty close for the first run. However, its not super accurate for some cases like outputting 7 etc. I am trying to figure why that is the case!

[Rohan2821999]

Since, this is a single layer network (single bias and weight) I am a bit unclear about how to figure out or show which neurons are active for specific numbers..

[cbattista]

Cool - I think it's time for me to dig into some code here. Which file in the repo is doing this - ImgNN.py, right?

On Fri, Sep 16, 2016 at 5:38 PM, Rohan Hundia notifications@github.com wrote:

Since, this is a single layer network (single bias and weight) I am a bit unclear about how to figure out or show which neurons are active for specific numbers..

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rohan2821999/Neural-Nets/issues/2#issuecomment-247738027, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOuOzsHr-QB5yN2sdHgUUXKNldMk2Ejks5qqzaUgaJpZM4J_dKm .

[Rohan2821999]

Yup! It's that one.. I am training the data directly on the xinput so all nodes are directly connected..

[Rohan2821999]

Figured out how Zorzi network works (Below are the layers of computation):

1. Layer1 Computation = Xinput matrix X Weight1 matrix + Bias matrix
2. Layer2 Computation = Layer1 output X Weight2 matrix + Bias matrix --> activation values

The above layers are trained according to contrastive divergence (which is basically a complicated name for averaged gradient descent) to give the desired activation values after 'n' epochs.

The activation values from above neural net are fed into a neural net (backpropogated and reconstructed)

Reconstruct1 = Activation matrix X Weight2 + Bias2 Reconstruct2 = Reconstruct1 X Weight1 + Bias1 --> desired output close to x values based on neurons

A separate neural net is needed for the above reconstructions since a separate number of epochs of training is required.

I have therefore mimicked exactly the zorzi's neural net however I am unclear about dimensions of matrices he has used (just copied from there to initially get it working). Also, I am unsure on how to run two separate neural nets nested under a single network (would look into that).

I think we could try to run it on Sherlock tomorrow (if possible)

[cbattista]

Great. Jonathan will give us a tour of Sherlock at 1pm tomorrow!

On Mon, Sep 19, 2016 at 5:07 PM, Rohan Hundia notifications@github.com wrote:

Figured out how Zorzi network works (Below are the layers of computation):

1. Layer1 Computation = Xinput matrix X Weight1 matrix + Bias matrix
2. Layer2 Computation = Layer1 output X Weight2 matrix + Bias matrix --> activation values

The above layers are trained according to contrastive divergence (which is basically a complicated name for averaged gradient descent) to give the desired activation values after 'n' epochs.

The activation values from above neural net are fed into a neural net (backpropogated and reconstructed)

Reconstruct1 = Activation matrix X Weight2 + Bias2 Reconstruct2 = Reconstruct1 X Weight1 + Bias1 --> desired output close to x values based on neurons

A separate neural net is needed for the above reconstructions since a separate number of epochs of training is required.

I have therefore mimicked exactly the zorzi's neural net however I am unclear about dimensions of matrices he has used (just copied from there to initially get it working). Also, I am unsure on how to run two separate neural nets nested under a single network (would look into that).

I think we could try to run it on Sherlock tomorrow (if possible)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rohan2821999/Neural-Nets/issues/2#issuecomment-248165115, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOuOzfMWuzTNoTjXYjNUE_sGyd3XDphks5qryPNgaJpZM4J_dKm .

[Rohan2821999]

@cbattista ... So, I built a generate Bayesian based neural net model where the weights and the hidden layers get updated from probability distributions of features generated from the Xinput (autonencoder RBMM based with 2 hidden layers).. It's a completely unsupervised model (unlike before which was kind of supervised) and works for unseen data as well.(works decently).

The weights are randomly initialized from a gaussian curve every time the model is run so the output is not always the same however, I could definitely figure out which values of initial weights yields the best output.

Below is the sample output of image produced:

Unsupervised model output:

Corresponding Xinput:

The spatial locations of not all circles match however, the number is more or less the same.

I would send you the histograms of hidden layers for specific numbers later today..

Do you think something like above would be good?

[cbattista]

Yeah, that looks pretty good.

On Thu, Sep 22, 2016 at 3:19 PM, Rohan Hundia notifications@github.com wrote:

@cbattista https://github.com/cbattista ... So, I built a generate Bayesian based neural net model where the weights and the hidden layers get updated from probability distributions of features generated from the Xinput (autonencoder RBMM based with 2 hidden layers).. It's a completely unsupervised model (unlike before which was kind of supervised) and works for unseen data as well.(works decently).

The weights are randomly initialized from a gaussian curve every time the model is run so the output is not always the same however, I could definitely figure out which values of initial weights yields the best output.

Below is the sample output of image produced:

Unsupervised model output:

[image: image] https://cloud.githubusercontent.com/assets/13100688/18768093/c6594730-80d7-11e6-864b-fd71f2ffe2d4.png

Corresponding Xinput: [image: image] https://cloud.githubusercontent.com/assets/13100688/18768058/a16072e6-80d7-11e6-8d43-c5b811873d87.png

The spatial locations of not all circles match however, the number is more or less the same.

I would send you the histograms of hidden layers for specific numbers later today..

Do you think something like above would be good?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rohan2821999/Neural-Nets/issues/2#issuecomment-249044891, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOuO1L9kouartG5gVk2uLv5SnlK7ZQyks5qsv7lgaJpZM4J_dKm .

[Rohan2821999]

Okay! So, I ran the network by training individual set of images (all types of 3 dots only). And then I checked which neurons activated the most - the neuron number is not all always the same. However, there are a set of neurons which when activated the most would yield 3 dots output.

For example, sometimes neuron 25 gets activated, other times neuron 45 gets activated etc. and when these specific neurons get activated it yields good results. However, though if neuron 22 is activated and backpropogated then the corresponding output image is bad..

[Rohan2821999]

In the above case, I am training only a set of number dot images (for example only 3 above) to get the histogram. I don't think that's a good way to do it - I think we should be training the entire dataset together but then I don't know how to get a histogram for specific numbers when the entire dataset is inputted.

[cbattista]

Yeah, you definitely need to train it on the entire dataset.

Then, once it's train, you just show it one image and get the activation in the hidden units for that particular image.

-C

On Thu, Sep 22, 2016 at 8:56 PM, Rohan Hundia notifications@github.com wrote:

In the above case, I am training only a set of number dot images (for example only 3 above) to get the histogram. I don't think that's a good way to do it - I think we should be training the entire dataset together but then I don't know how to get a histogram for specific numbers when the entire dataset is inputted.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rohan2821999/Neural-Nets/issues/2#issuecomment-249096819, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOuO0tiuedehrKusORajUKLh3XozPquks5qs04LgaJpZM4J_dKm .

[cbattista]

*trained

On Thu, Sep 22, 2016 at 9:39 PM, Christian Battista < battista.christian@gmail.com> wrote:

Yeah, you definitely need to train it on the entire dataset.

Then, once it's train, you just show it one image and get the activation in the hidden units for that particular image.

-C

On Thu, Sep 22, 2016 at 8:56 PM, Rohan Hundia notifications@github.com wrote:

In the above case, I am training only a set of number dot images (for example only 3 above) to get the histogram. I don't think that's a good way to do it - I think we should be training the entire dataset together but then I don't know how to get a histogram for specific numbers when the entire dataset is inputted.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rohan2821999/Neural-Nets/issues/2#issuecomment-249096819, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOuO0tiuedehrKusORajUKLh3XozPquks5qs04LgaJpZM4J_dKm .

[Rohan2821999]

I am not sure how to do that. I can train the entire data set but I am not sure how to just show it one image and get the activation.

[Rohan2821999]

The local minima issue is not resolved by the workaround we thought about.. Once it gets stuck changing the step is not making a difference and nor is randomly changing a given set of weights..

So, I hold some ambiguity whether it really is a local minima issue or something else!