Multinode output and hidden layer output

[saj1919]

Hi, I am new to keras and trying to learn. I want to create and train neural network with 5 node input layer, 16 node hidden layer and 5 node output layer. Input and output are same. After training want to discard output layer and then hidden layer should output. (AutoEncoder problem ??)

          * * * * *                   <- Input layer

* * * * * * * * * * * * * * * *       <- hidden layer

          * * * * *                   <- Output layer

Couldn't find any example related to it where output layer has more than 1 node output ( went though otto and imdb examples) Can I get some example or go ahead ?

[lukedeo]

You're going to want to look at the AutoEncoder class. Note that this isn't tested. You could do something like:

ae = Sequential()
ae.add(
    AutoEncoder(
        encoder=Dense(nb_inputs, nb_hidden, activation='relu'),
        decoder=Dense(nb_hidden, nb_inputs, activation='relu'), 
        output_reconstruction=False, 
        tie_weights=True
        )
    )
ae.compile(loss='mse', optimizer=Adam())
ae.fit(X, X)

# -- ae.predict will return the encoding (hidden layer) since we set output_reconstruction=False
R = ae.predict(X)

Now, if you wanted to use the encoder in another model, you could:

ff = Sequential()
ff.add(ae[0].encoder)
ff.add(Dense(nb_hidden, nb_outputs))
# ... do whatever else

[mthrok]

That's exactly the autoencoder pattern.

See https://github.com/fchollet/keras/issues/358#issuecomment-119379780

You need to set output_reconstruction=Falseafter training.

[fchollet]

The input to Keras models is numpy arrays (or sometimes lists of numpy arrays). Not Python lists. Convert your lists to numpy arrays and it will work.

On 8 July 2015 at 22:52, Swapnil Jadhav notifications@github.com wrote:

Thanks for the quick reply. What I am trying to do is train model with training data 1 to 65535 (70% randomly chosen) and hidden layer of 16 nodes and then check for remaining 30% numbers .. what representation they get. (Example I thought of learning keras and unsupervised nn .. don't know I am hoping to get binary representation for numbers !!).

Here is the code -

from random import randint

import logging logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

def gen_input(): Xtrain = [] Xtest = [] for i in range(65536): tmp = randint(0,9) if tmp<=7: Xtrain.append(i) else: Xtest.append(i)

X_train = []
for x in Xtrain:
x_str = str(x)
x_list = []
if len(x_str) < 5:
    n = 5-len(x_str)
    for i in range(n):
       x_list.append(0)
for x_char in x_str:
    x_list.append(int(x_char))
X_train.append(x_list)

X_test = []
for x in Xtest:
x_str = str(x)
x_list = []
if len(x_str) < 5:
    n = 5-len(x_str)
    for i in range(n):
       x_list.append(0)
for x_char in x_str:
    x_list.append(int(x_char))
X_test.append(x_list)

return X_train, X_test

MAIN

from keras.layers.core import Dense, AutoEncoder from keras.models import Sequential from keras.optimizers import RMSprop

X_train, X_test = gen_input() X_train = X_train X_test = X_test

print "Len X_train : "+str(len(X_train)) print "Len X_train[0] : "+str(len(X_train[0])) print "Len X_test : "+str(len(X_test)) print "Len X_test[0] : "+str(len(X_test[0])) print print "Type X_train : "+str(type(X_train)) print "Type X_train[0] : "+str(type(X_train[0])) print "Type X_test : "+str(type(X_test)) print "Type X_test[0] : "+str(type(X_test[0]))

batch_size = 32 nb_epoch = 1

model = Sequential() model.add(AutoEncoder(encoder=Dense(5, 16), decoder=Dense(16, 5), output_reconstruction=True, tie_weights=True))

model.compile(loss='mean_squared_error', optimizer=RMSprop()) model.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1, validation_data=[X_test, X_test])

all_weights = [] for layer in model.layers: w = layer.get_weights() all_weights.append(w) print w print

Getting following error :

/usr/bin/python2.7 /home/saj1919/PycharmProjects/NL_Subject_Analyzer/src/article_clf/keras_trial.py Len X_train : 52645 Len X_train[0] : 5 Len X_test : 12891 Len X_test[0] : 5

Type X_train : <type 'list'> Type X_train[0] : <type 'list'> Type X_test : <type 'list'> Type X_test[0] : <type 'list'> Traceback (most recent call last): File "/home/saj1919/PycharmProjects/NL_Subject_Analyzer/src/article_clf/keras_trial.py", line 69, in model.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1, validation_data=[X_test, X_test]) File "build/bdist.linux-x86_64/egg/keras/models.py", line 371, in fit File "build/bdist.linux-x86_64/egg/keras/models.py", line 129, in _fit File "build/bdist.linux-x86_64/egg/keras/models.py", line 42, in slice_X TypeError: only integer arrays with one element can be converted to an index Train on 5 samples, validate on 5 samples Epoch 0

Process finished with exit code 1

— Reply to this email directly or view it on GitHub https://github.com/fchollet/keras/issues/365#issuecomment-119828451.

[saj1919]

Thanks . Now I am getting results like following - [ 0. 0. 0. 0. 3.] : [-0.00438542 -0.00509626 -0.00570252 -0.00357726 2.99103961] [ 0. 0. 0. 0. 4.] : [-0.00628674 -0.00774946 -0.00816605 -0.00497658 3.98780408] [ 0. 0. 0. 0. 6.] : [-0.01008939 -0.01305586 -0.0130931 -0.00777521 5.98133302] [ 0. 0. 0. 0. 7.] : [-0.01199071 -0.01570906 -0.01555663 -0.00917453 6.97809748] [ 0. 0. 0. 1. 6.] : [-0.00957261 -0.01211282 -0.01196831 0.99251705 5.98243663]

I want to get results from hidden layer. Is there way to do that? I tried to set "output_reconstruction=False" to get array of 16 as output ... but do not seem to work, I am getting array of 5 as output.

My current code -

from keras.layers.core import Dense, AutoEncoder
from keras.models import Sequential
from keras.optimizers import RMSprop
import numpy as np

X_train, X_test = gen_input()
X_train = np.asarray(X_train).astype("float32")
X_test = np.asarray(X_test).astype("float32")

batch_size = 32
nb_epoch = 10

model = Sequential()
model.add(AutoEncoder(encoder=Dense(5, 16), decoder=Dense(16, 5), output_reconstruction=True, tie_weights=True))

model.compile(loss='mean_squared_error', optimizer=RMSprop())
model.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1, validation_split=0.1)
model.__setattr__('output_reconstruction', False)

X_pred = model.predict(X_test)
for i in range(5):
    print str(X_test[i]) +" : "+ str(X_pred[i])

[saj1919]

My bad .. I was putting validation and making output_reconstruction=False resulting in error. Now I am getting what I wanted. Many Thanks !!

model = Sequential()
model.add(AutoEncoder(encoder=Dense(5, 16), decoder=Dense(16, 5), output_reconstruction=False, tie_weights=True))
model.compile(loss='mean_squared_error', optimizer=RMSprop())
model.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1)

Also, is there way of using unsupervised learning method (like clustering, pattern recognition) in keras. (Studying this -> http://deeplearning.stanford.edu/wiki/index.php/Self-Taught_Learning) Any examples will be appreciated.

[fchollet]

Also, is there way of using unsupervised learning method (like clustering, pattern recognition) in keras.

Yes, if you can define an appropriate supervised task to go with it, you can do it in Keras. All "unsupervised" learning is actually supervised (by a cost function).