About implementation detail

[yasunorikudo]

How does this code predict heatmap? Let me know about referenced paper if you can.

[pengpaiSH]

Same question. How did you train these networks for heatmap?

[leonardblier]

Hi, thank you for your question.

The networks are not trained to produce a heatmap, it's a trick that can be used with an already trained CNN. Let's take as an example the AlexNet, which take an input of size (227,227), and the dog picture, which is of size 2560x1600. Then we compute the output of the network for each sub-frame of size (227,227) of the entire picture. This produces for each one a score (for each label), and this is our heatmap, of size (2560 - 227, 1600 - 227) = (2333, 1373).

But just computing the output for each sub-frame would be computionally heavy. So what we do instead is that we convert the fully-connected layers into convolutional layers, only by reshaping the weights matrix, so this can be computed quite fast. With the different sub-samples at the different scales, the final output shape is (73, 43)

Maybe to understand well, you can look at the end of the definition of a CNN, AlexNet as an example :

   if heatmap:
        dense_1 = Convolution2D(4096,6,6,activation="relu",name="dense_1")(dense_1)
        dense_2 = Convolution2D(4096,1,1,activation="relu",name="dense_2")(dense_1)
        dense_3 = Convolution2D(1000, 1,1,name="dense_3")(dense_2)
        prediction = Softmax4D(axis=1,name="softmax")(dense_3)
    else:
        dense_1 = Flatten(name="flatten")(dense_1)
        dense_1 = Dense(4096, activation='relu',name='dense_1')(dense_1)
        dense_2 = Dense(4096, activation='relu',name='dense_2')(dense_2)
        dense_3 = Dense(1000,name='dense_3')(dense_3)
        prediction = Activation("softmax",name="softmax")(dense_3)

This has been made before, like here, in caffe (at the end) : https://github.com/BVLC/caffe/blob/master/examples/net_surgery.ipynb It has also be used in academic papers to improve classification, like here : Thibaut Durand, Nicolas Thome, Matthieu Cord, ICCV2015 But I don't know when it has been done for the first time.

[pengpaiSH]

@leonardblier Thank you for your quick response! Now I finally understand that the networks for producing heatmaps are not trained from scratch. What still confuses me is that (1) how could you "reshape" the AlexNet simply by

        dense_1 = Convolution2D(4096,6,6,activation="relu",name="dense_1")(dense_1)
        dense_2 = Convolution2D(4096,1,1,activation="relu",name="dense_2")(dense_1)
        dense_3 = Convolution2D(1000, 1,1,name="dense_3")(dense_2)
        prediction = Softmax4D(axis=1,name="softmax")(dense_3)

while loading the pretrained weights? Not that these weights are for the original AlexNets, i.e. with fully connected layers instead of convolutional layers in the last two layers.

(2) What's the Softmax4D purpose? I cannot figure it out from the source codes.

[leonardblier]

@paipai880429 You're welcome. 1) The piece of code you are quoting is indeed only defining the model. But to load the weights, it loads a model designed to produce a heatmap, and an other normal. It loads the weights for this last one, and then reshape it for the new network. Here is a script inspired by the function convnet from convnet.py, that does almost the same thing :

convnet = Alexnet(weights_path="weights/alexnet_weights.h5", heatmap=False)
convnet_heatmap = Alexnet(heatmap=True)
for layer in convnet_heatmap.layers:
    if layer.name.startswith("conv"):
        orig_layer = convnet.get_layer(layer.name)
        layer.set_weights(orig_layer.get_weights())
    elif layer.name.startswith("dense"):
        orig_layer = convnet.get_layer(layer.name)
        W,b = orig_layer.get_weights()
        n_filter,previous_filter,ax1,ax2 = layer.get_weights()[0].shape
        new_W = W.reshape((previous_filter,ax1,ax2,n_filter))
        new_W = new_W.transpose((3,0,1,2))
        new_W = new_W[:,:,::-1,::-1]
        layer.set_weights([new_W,b])
return convnet_heatmap

2) The Softmax4D is just a softmax activation layer, applying softmax along a given axis. We had to do this because the softmax layer in Keras is only for tensors of dimension 2 or 3 (I don't figure out why ...), and ours are of dimension 4.

[pengpaiSH]

@leonardblier Thank you for your patience. One more question is that where does 6,6 and 1,1 in the Convolution2D come from?

        dense_1 = Convolution2D(4096,6,6,activation="relu",name="dense_1")(dense_1)
        dense_2 = Convolution2D(4096,1,1,activation="relu",name="dense_2")(dense_1)

[michaelosthege]

hi there, I found this while looking for a (keras) implementation of Fast Image Scanning with Deep Max-Pooling Convolutional Neural Networks (A. Giusti et al. 2013) and have a few questions. Please correct me if I'm wrong:

1. the heatmap resolution, depends on the spatial shape of the input layer to the FC ones, right? for example of the spatial size of the last maxpooling is 1x1, the windows will not overlap. if it's 2x2 they will overlap by 50 % and so on
2. Giusti et al. demonstrated a way to get an output corresponding to a sliding window stride of 1 pixel by basically doing all possible maxpooling offsets in parallel and then taking a subset of the resulting outputs. however they do not describe how to assign the resulting maps to the input pixels. This is the only implementation of it I found so far, but it's based on numpy. Anyone else interested in a keras implementation of this?

I also don't get how Atrous convolution relates to this. cheers, michael

[thiippal]

Hi @leonardblier, I'm using your code to reshape the weights of my pre-trained TensorFlow model, but have run into a problem described here.

Basically, I want to turn the last two dense layers into convolutional ones: for the first ones, the weights are loaded, but for the second set I get the following error using the model.load_weights() function in Keras:

ValueError: Cannot feed value of shape (512, 3) for Tensor u'Placeholder_10:0', which has shape '(1, 1, 512, 3)'

Any ideas how to fix this? I'm using the TensorFlow backend with Keras.

[michaelosthege]

Yes, the weights have to be reshaped. I wrote a function to convert a sequential model with a dense classifier into a fully-convolutional one.

You just have to load the original trained model and feed it to the function. It will return a new model that is fully convolutional. The weights are copied and reshaped from the original model.

No guarantees though ;)

def makeExpandedConvnet(patchnet, inputShape):
    """Takes a model with convolution layers followed by fully-connected layers and constructs
    a bigger network that takes bigger input. The fully connected layers are converted into Convolution2D
    layers such that it becomes equivalent to sliding a window.

    Parameters
    ----------
    patchnet : keras.models.Sequential
        a sequential model of a Conv2D/Pool2D feature extraction, followed by a Dense classifier/regressor

    inputShape : tuple
        shape (c,h,w) of the big input image

    Returns
    -------
    convnet : keras.models.Sequential
        a fully-convolutional sequential model, outputting a 2D-map (matrix) of the prediction

    patchshape : tuple
        shape (c,h,w) of the "sliding window"

    Reference
    ---------
    https://github.com/heuritech/convnets-keras
    """
    import keras
    from .. engine import layers
    convnet = keras.models.Sequential()
    patchshape = None
    featureMapsShape = None
    encounteredFirstDense = False
    for l,layer in enumerate(patchnet.layers):
        type = layer.__class__
        if (l == 0):                                                                # input has a different shape
            convnet.add(keras.layers.InputLayer(inputShape, name=layer.name))
            patchshape = layer.input_shape[1:]
        elif (type is keras.layers.Flatten):                                       # skip flattening
            featureMapsShape = layer.input_shape
            featureExtractionLayersCount = len(convnet.layers)
        elif (type is keras.layers.Dense):
            isFirstDenseLayer = encounteredFirstDense == False
            if (isFirstDenseLayer):
                encounteredFirstDense = True
            # first collect the configuration of the convolutional dense layers
            nb_units = layer.output_dim
            if isFirstDenseLayer:    # the first convolutional dense layer may take spatial feature maps as input
                nb_row, nb_col = (featureMapsShape[-2], featureMapsShape[-1])                
            else:
                nb_row, nb_col = (1,1)                                   # later convolutional dense layers don't take spatial information
            # create layer, copy activation function (may be softmax), copy weights
            activation = layer.activation if layer.activation.__name__ != 'softmax' else 'linear'
            newlayer = keras.layers.Convolution2D(nb_units, nb_row, nb_col, activation=activation, subsample=(1,1), name=layer.name)
            convnet.add(newlayer)
            # reshape and load weights as seen here: https://github.com/heuritech/convnets-keras/blob/master/convnetskeras/convnets.py
            W,b = layer.get_weights()   # trained weights of Dense layer
            n_filter,previous_filter,ax1,ax2 = newlayer.get_weights()[0].shape  # target shape of Conv Dense layer
            new_W = W.reshape((previous_filter,ax1,ax2,n_filter))
            new_W = new_W.transpose((3,0,1,2))
            new_W = new_W[:,:,::-1,::-1]
            newlayer.set_weights([new_W,b])
            # if we encountered a softmax-activated layer, we must append a 4D softmax on top of a linearly activated Conv2D
            if (layer.activation.__name__ == 'softmax'):
                convnet.add(layers.Softmax4D(axis=1))
        else:                                                                       # copy feature extraction and activation layers
            config = {'class_name': type, 'config': layer.get_config()}
            newlayer = keras.utils.layer_utils.layer_from_config(config)    # copy design from patchnet
            convnet.add(newlayer)
            newlayer.set_weights(layer.get_weights())                       # copy weights from patchnet
    # the convnet is now finished. it will output a probability map
    return convnet, patchshape

The layers.Softmax4D() is the one from convnetskeras\customlayers.py

[chi-hung]

Hi,

I'm using the 'SoftmaxMap' of your code (I've modified it slightly in accordance with the Keras Documentation):

from keras.engine import Layer
import keras.backend as K

class SoftmaxMap(Layer):
    # Init function
    def __init__(self, axis=-1, **kwargs):
        self.axis = axis
        super(SoftmaxMap, self).__init__(**kwargs)
    def build(self,input_shape):
        pass
    def call(self, x, mask=None):
        e = K.exp(x - K.max(x, axis=self.axis, keepdims=True))
        s = K.sum(e, axis=self.axis, keepdims=True)
        return e / s
    def compute_output_shape(self, input_shape):
        return input_shape

I'm doing this because I'd like to run a fully convolutional network with Keras (see the following for the network structure)

model = Sequential()

#conv1
model.add(Conv2D(filters=96, kernel_size=(11, 11),
                 strides=(4,4),
                 padding='valid',
                 input_shape=(None,None,3)
                )
         )
model.add(Activation('relu'))
#pooling1
model.add(MaxPooling2D(pool_size=(3, 3),
                       strides=(2,2),
                       padding='valid'
                      )
         )
#conv2
model.add(Conv2D(filters=256, kernel_size=(5, 5),
                 strides=(1,1),
                 padding='same'
                )
         )
model.add(Activation('relu'))
#pooling2
model.add(MaxPooling2D(pool_size=(3, 3),
                       strides=(2,2),
                       padding='valid'
                      )
         )
#conv3
model.add(Conv2D(filters=384, kernel_size=(3, 3),
                 strides=(1,1),
                 padding='same'
                )
         )
model.add(Activation('relu'))
#conv4
model.add(Conv2D(filters=384, kernel_size=(3, 3),
                 strides=(1,1),
                 padding='same'
                )
         )
model.add(Activation('relu'))
#conv5
model.add(Conv2D(filters=256, kernel_size=(3, 3),
                 strides=(1,1),
                 padding='same'
                )
         )
model.add(Activation('relu'))
#pooling3
model.add(MaxPooling2D(pool_size=(3, 3),
                       strides=(2,2),
                       padding='valid'
                      )
         )
#conv6
model.add(Conv2D(filters=4096, kernel_size=(6, 6),
                 strides=(1,1),
                 padding='valid'
                )
         )
model.add(Activation('relu'))
#conv7
model.add(Conv2D(filters=4096, kernel_size=(1, 1),
                 strides=(1,1),
                 padding='valid'
                )
         )
model.add(Activation('relu'))
#conv8
model.add(Conv2D(filters=2, kernel_size=(1, 1),
                 strides=(1,1),
                 padding='valid'
                )
         )
#model.add(Flatten())
#model.add(GlobalAveragePooling2D())
#model.add(Activation('softmax'))

model.add(SoftmaxMap(axis=-1))

model.compile(loss='categorical_crossentropy',
              optimizer=SGD(lr=0.005,momentum=0.3),
              metrics=['accuracy'])

However, I got an error while fitting the model:

ValueError: Error when checking target: expected softmax_map_1 to have 4 dimensions, but got array with shape (128, 2)

which is quite weird. I'm using Tensorflow 1.2.1. Any suggestions?

[michaelosthege]

There is a mismatch between the output shape of the last layer and the labels. You should look closely at the input and output shapes of the softmax layer.

From: chi-hungmailto:notifications@github.com Sent: Freitag, 11. August 2017 06:52 To: heuritech/convnets-kerasmailto:convnets-keras@noreply.github.com Cc: michaelosthegemailto:thecakedev@hotmail.com; Commentmailto:comment@noreply.github.com Subject: Re: [heuritech/convnets-keras] About implementation detail (#1)

Hi,

I'm using the 'SoftmaxMap' of your code (I've modified it slightly in accordance with the Keras Documentationhttps://keras.io/layers/writing-your-own-keras-layers/):

from keras.engine import Layer import keras.backend as K

class SoftmaxMap(Layer):

Init function

def __init__(self, axis=-1, **kwargs):
    self.axis = axis
    super(SoftmaxMap, self).__init__(**kwargs)
def build(self,input_shape):
    pass
def call(self, x, mask=None):
    e = K.exp(x - K.max(x, axis=self.axis, keepdims=True))
    s = K.sum(e, axis=self.axis, keepdims=True)
    return e / s
def compute_output_shape(self, input_shape):
    return input_shape

I'm doing this because I'd like to run a fully convolutional network with Keras (see the following for the network structure)

model = Sequential()

conv1

model.add(Conv2D(filters=96, kernel_size=(11, 11), strides=(4,4), padding='valid', input_shape=(None,None,3) ) ) model.add(Activation('relu'))

pooling1

model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2), padding='valid' ) )

conv2

model.add(Conv2D(filters=256, kernel_size=(5, 5), strides=(1,1), padding='same' ) ) model.add(Activation('relu'))

pooling2

model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2), padding='valid' ) )

conv3

model.add(Conv2D(filters=384, kernel_size=(3, 3), strides=(1,1), padding='same' ) ) model.add(Activation('relu'))

conv4

model.add(Conv2D(filters=384, kernel_size=(3, 3), strides=(1,1), padding='same' ) ) model.add(Activation('relu'))

conv5

model.add(Conv2D(filters=256, kernel_size=(3, 3), strides=(1,1), padding='same' ) ) model.add(Activation('relu'))

pooling3

model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2), padding='valid' ) )

conv6

model.add(Conv2D(filters=4096, kernel_size=(6, 6), strides=(1,1), padding='valid' ) ) model.add(Activation('relu'))

conv7

model.add(Conv2D(filters=4096, kernel_size=(1, 1), strides=(1,1), padding='valid' ) ) model.add(Activation('relu'))

conv8

model.add(Conv2D(filters=2, kernel_size=(1, 1), strides=(1,1), padding='valid' ) )

model.add(Flatten())

model.add(GlobalAveragePooling2D())

model.add(Activation('softmax'))

model.add(SoftmaxMap(axis=-1))

model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.005,momentum=0.3), metrics=['accuracy'])

However, I got an error while fitting the model:

ValueError: Error when checking target: expected softmax_map_1 to have 4 dimensions, but got array with shape (128, 2)

which is quite weird. I'm using Tensorflow 1.2.1. Any suggestions?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/heuritech/convnets-keras/issues/1#issuecomment-321731703, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFnx8qo8aeVO5tmEPKqwUY3f6hWkhHuQks5sW94fgaJpZM4IQH6e.

[chi-hung]

@michaelosthege

Thank you for the very useful comment. I've fixed my code and now it works as expected! I was using flow_from_directory for data augmentation, and didn't notice about the label format out of the generator...