How to classify my own data using the pre-trained model directly ?

[iiwindii]

Hi guys, how to evaluate the pre-trained model directly using my own data? Certainly I can use the pre-trained model (e.g. fully-connected layer) to extract features and then train a svm or softmax classifier. But how to implement this process in an end-to-end way just like the process when we train a model during which see the top-1 and top-5 error. In other words, can I use the pre-trained model for classification of my own data directly?

[iiwindii]

@vedaldi hi, could you give some advice? Thanks in advance!

[arunmallya]

You can load the pretrained network .mat file, change the last conv/fc layer to reflect the new classification/regression task at hand. For example, take the pretrained VGG_VD-16 network, remove the last layer and replace it with what you want, say a 200-way classifier as follows:

function net = create_net_from_pretrained(pretrainedPath)
    net = []; % your network structure

    % load pre-trained network
    old_net = load(pretrainedPath);

    % add dropout layers in network (saved model has dropout removed)
    drop1 = struct('name', 'dropout6', 'type', 'dropout', 'rate' , 0.5) ;
    drop2 = struct('name', 'dropout7', 'type', 'dropout', 'rate' , 0.5) ;
    old_net.layers = [old_net.layers(1:33) drop1 old_net.layers(34:35) drop2 old_net.layers(36:end)] ;

    % ignore classification and last softmax layers (we will insert our own)
    net.layers = old_net.layers(1:end-2);

    % add our own conv layer and loss layer
    % I use the add_block() function available in matconvnet/examples/cnn_imagenet_init.m
    net = add_block(net, opts, '8', 1, 1, 4096, 200, 1, 0) ; % 200-way fully connected layer fc8
    net.layers(end) = [] ; % remove ReLU layer that gets added after conv layer
    net.layers{end+1} = struct('type', 'softmaxloss') ; % add loss layer
end

You can then pass this net to cnn_train() after writing your own get_batch() function. Have a look at the files in examples/ and modify them accordingly.

[iiwindii]

@arunmallya hi, thanks for your response. But I think the answer is a bit different from my problem. Your answer is the way to fine-tune the pre-trained model on one's own data. My problem is how to classify my own data through the pretrained model directly, just like transfer learning, (For example, I can use VGG_VD-16 network to extract features and then the features can be used to train a svm or softmax classifier.), but without the training or fine-tune process. what I want to do is: given a new dataset (such as 21 classes), the pre-trained model can be used to classify them directly in an end-to-end way without training process.

[arunmallya]

I'm not quite sure what you mean by "end-to-end", because that itself suggests back-propagating from the new loss layer back to the conv layers. Yes, you can extract features by making a forward pass and extracting res(layer_number).x to get the features and then train an SVM. That's like attaching a new layer and changing its weights while keeping the rest of the network weights fixed.

[iiwindii]

@arunmallya thanks for help. In the above code, you have added the dropout layers. Is it necessary to do this while using the pre-trained model for fine-tune? I have fine-tuned the pre-trained models on my data set for some tasks, but the performance is not very good. I have not added the dropout layers...

[arunmallya]

Dropout might help. I have always used them after my fully connected layers while finetuning. Setting a slightly lower learning rate during finetuning might help, say 10 times lesser than the original network. Also, it might be a good idea to set larger learning rate multiplier for the newly inserted layers -> net.layers{id}.learningRate = [10, 20], while older layers have the default net.layers{id}.learningRate = [1, 2], or net.layers{id}.learningRate = [0, 0] if you want to keep them fixed. Doing the latter might be useful if you have very little training data and do not want to overfit.

[lyltencent]

@fengyunxiaozi , Hi, I am not sure you have found the solution yet. But here is what I did.

net = load('imagenet-caffe-ref.mat'); im = single(oriImg) ; % note: 255 range im = imresize(im, net.meta.normalization.imageSize(1:2)) ; im = im_ - net.meta.normalization.averageImage ; res = vlsimplenn(net, im) ;

   % viesion: matconvnet-1.0-beta17
   featVec = res(20).x;

   featVec = featVec(:);

[Ashutosh1995]

Sir, could you let me know how to extract the 4k dimensional feature if I am using a dataset of say 29,000 images by using Vgg Face model?