Can I visualize hidden layer?

[dikien]

I try to visualize hidden layers MNIST data. Below is code sample:

nn = Classifier(
    layers=[
        Layer("Sigmoid", units=300),
        Layer("Softmax")],
    learning_rate=0.01,
    n_iter=10)
nn.fit(trainX, trainY)

myae = ae.AutoEncoder(
            layers=[
            ae.Layer("Tanh", units=784),
            ae.Layer("Tanh", units=300),
            ae.Layer("Tanh", units=784),],
            learning_rate=0.002,
            n_iter=10)

scikit-neuralnetwork support this??

ps, I have many trouble with pylearn2 because it is not easy to use. Thank you.

[alexjc]

It's supported by accessing the data you need directly. Look at the functions called get_weights() in mlp.py, or get_biases() then visualize those.

I'm not sure we plan on adding any further support in the core code, but if you get it working you can contribute an example.

[ssamot]

You could do something like [(l.get_weights(), l.get_biases()) for l in nn.mlp.layers] and get the weights and the biases

[alexjc]

Closing this issue. If you want to contribute a visualization example, please submit a pull request!

[dikien]

Thank you for your comments. I figure out how to get weights and biases.

As I said, I try to do visualize hidden layers MNIST data. I figure out how to do it with Keras. https://github.com/dikien/break-captcha/blob/master/Step4_Generate_Clssifier_keras_conv_visualize.ipynb

However, I can't put those weights and biases into new layer. Is this bug or my problem? I uploaded my ipython notebook. (https://github.com/dikien/personnel-study/blob/master/deep-learning/scikit-neuralnetwork-study/q1-set-weights.ipynb)

Train first model

clf = Classifier(
        layers=[Layer("Rectifier", units=300), Layer("Softmax")],
        learning_rate=0.02,
        batch_size=100,
        n_iter=2,
        verbose=1,
)
clf.fit(X_train, y_train)

Make second model without train data.

l = Layer("Rectifier", units=300)
clf1 = Classifier(layers=[l])

Put weights and biases of first model into secede model. However, it occurs errors, I don't know why.

inf = [(l.get_weights(), l.get_biases()) for l in clf.mlp.layers]

clf1.mlp.layers[0].set_weights(inf[0][0])
clf1.mlp.layers[0].set_biases(inf[0][1])

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-93bc00c0264c> in <module>()
----> 1 clf1.mlp.layers[0].set_weights(inf[0][0])
      2 clf1.mlp.layers[0].set_biases(inf[0][1])

AttributeError: 'NoneType' object has no attribute 'layers'

Anyway, I tried to get output.

clf1.predict(X_test)

WARNING:sknn:Computing estimates with an untrained network.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-d876bb2f91fc> in <module>()
----> 1 clf1.predict(X_test)

/Users/dikien/anaconda/lib/python2.7/site-packages/scikit_neuralnetwork-0.2-py2.7.egg/sknn/mlp.pyc in predict(self, X)
    508         """
    509         y = self.predict_proba(X)
--> 510         return self.label_binarizer.inverse_transform(y, threshold=0.5)

/Users/dikien/anaconda/lib/python2.7/site-packages/sklearn/preprocessing/label.pyc in inverse_transform(self, Y, threshold)
    369         of inverse_transform.
    370         """
--> 371         self._check_fitted()
    372 
    373         if threshold is None:

/Users/dikien/anaconda/lib/python2.7/site-packages/sklearn/preprocessing/label.pyc in _check_fitted(self)
    287     def _check_fitted(self):
    288         if not hasattr(self, "classes_"):
--> 289             raise ValueError("LabelBinarizer was not fitted yet.")
    290 
    291     def fit(self, y):

ValueError: LabelBinarizer was not fitted yet.

I tried again put them into. There is no error. I don't know why...

clf1.mlp.layers[0].set_weights(inf[0][0])
clf1.mlp.layers[0].set_biases(inf[0][1])

predict again and show up error again.

clf1.predict(X_test)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-d876bb2f91fc> in <module>()
----> 1 clf1.predict(X_test)

/Users/dikien/anaconda/lib/python2.7/site-packages/scikit_neuralnetwork-0.2-py2.7.egg/sknn/mlp.pyc in predict(self, X)
    508         """
    509         y = self.predict_proba(X)
--> 510         return self.label_binarizer.inverse_transform(y, threshold=0.5)

/Users/dikien/anaconda/lib/python2.7/site-packages/sklearn/preprocessing/label.pyc in inverse_transform(self, Y, threshold)
    369         of inverse_transform.
    370         """
--> 371         self._check_fitted()
    372 
    373         if threshold is None:

/Users/dikien/anaconda/lib/python2.7/site-packages/sklearn/preprocessing/label.pyc in _check_fitted(self)
    287     def _check_fitted(self):
    288         if not hasattr(self, "classes_"):
--> 289             raise ValueError("LabelBinarizer was not fitted yet.")
    290 
    291     def fit(self, y):

ValueError: LabelBinarizer was not fitted yet.

[alexjc]

I'm not sure what you're trying to do with setting the weights, but see the self.weights variable that's used for pickling.

You probably need to run _initialize() before setting the weights, or the layers are None.

[dikien]

I am trying to get output of each layers. Below is code written Keras style:

# Creating the model which consists of 3 conv layers followed by
# 2 fully conntected layers

# Sequential wrapper model
model = Sequential()

# first convolutional layer
model.add(Convolution2D(32, 1, 2, 2))
model.add(Activation('relu'))

# second convolutional layer
model.add(Convolution2D(48, 32, 2, 2))
model.add(Activation('relu')) 
model.add(MaxPooling2D(poolsize=(2,2)))

# third convolutional layer
model.add(Convolution2D(32, 48, 2, 2))
model.add(Activation('relu'))
model.add(MaxPooling2D(poolsize=(2,2)))

# convert convolutional filters to flatt so they can be feed to 
# fully connected layers
model.add(Flatten())

# first fully connected layer
model.add(Dense(32*6*6, 144, init='lecun_uniform'))
model.add(Activation('relu'))
model.add(Dropout(0.5))

# second fully connected layer
model.add(Dense(144, 144, init='lecun_uniform'))
model.add(Activation('relu'))
model.add(Dropout(0.5))

# last fully connected layer which output classes
model.add(Dense(144, 9, init='lecun_uniform'))
model.add(Activation('softmax'))

# setting sgd optimizer parameters
sgd = SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd)

model.fit(trainX, trainY,
          batch_size=batch_size,
          nb_epoch=nb_epoch,
          show_accuracy=True,
          verbose=1,
          validation_data=(testX, testY))

Make a new model for first Convolutional Layer and add weights values from original model.

model1 = Sequential()
model1.add(Convolution2D(32, 1, 2, 2, weights=model.layers[0].get_weights()))
model1.add(Activation('relu'))
model1.compile(loss='categorical_crossentropy', optimizer=sgd)

It shows output of first layer.

valX_h_1 = model1.predict(valX)

Make second model for second Convolutional Layer and add weights values from original model.

model2 = Sequential()
model2.add(Convolution2D(48, 32, 2, 2, weights=model.layers[2].get_weights()))
model2.add(Activation('relu'))
model2.compile(loss='categorical_crossentropy', optimizer=sgd)

It shows output of second layer.

valX_h_2 = model2.predict(valX_h_1)

This is what I am trying to do in scikit-neuralnetwork.

[alexjc]

Look at the transfer() function of the auto-encoder (ae.py) and see how it stores the weights inside an MLP that can be used during initialization. This works already, but I will talk to @ssamot to see if we should add a public interface for this...

[zmatilsky901]

hey guys, I'm also working on the MNIST dataset and was also wondering if there was a way to get the weight vectors at each layer of my neural net. In the discussion above, @alexjc seems to suggest a function called get_weights() that can be called on a particular layer object, but I see no such function. Was this functionality removed or something? This is some simple sample code to show you an example of what I'm looking for:

nn = Classifier(
    layers=[
        Layer("Sigmoid", units=100),
        Layer("Softmax")],
    learning_rate=0.001,
    n_iter=2)
nn.fit(train_set[0], train_set[1])

x = nn.layers[0]
x.get_weights()

Running this code returns the following error:

AttributeError                            Traceback (most recent call last)
 in ()
      1 x = nn.layers[0]
----> 2 x.get_weights()

AttributeError: 'Layer' object has no attribute 'get_weights'

As a final note, I see that my neural net (defined above) as "nn" does have an attribute called "weights", but this attribute returns nothing. Any suggestions are welcome.

[holderm]

I'm also interested in this. So I run the sample code given above and I obtain this error: AttributeError: 'Classifier' object has no attribute 'mlp' Can anyone suggests a solution?

[alexjc]

It's easiest to call __getstate__ on the MLP, which returns a dictionary including a long list of (weights, biases) as numpy arrays. We'll add a better function for this in upcoming releases!

See the code, the same used during pickling: https://github.com/aigamedev/scikit-neuralnetwork/blob/master/sknn/mlp.py#L86