convolutional neural networks example always returns the same output

[Tomue]

I am experimenting with the mnist and the cifar example of MatConvNet to get started with the area of convolutional neural networks. Until now I just trained the networks. Now I wanted to use them to classify images in the same way I used the pre-trained networks form MatConvNet. To use the network I wrote the following script. But no matter which image I put in the network it returns always the same answer. If anybody now an answer to my problem please let me know. Thanks

[Tomue]

`%% init oldFolder = cd; run setup; cd(oldFolder);

% load the pre-trained CNN net = load('cifar_cnn.mat'); net.layers(end)=[];

%% load and preprocess an image

im = imread('horse10.png') ; figure(1) subplot(2,2,1), imagesc(im); title('image to classify');

im_sing = im2single(im); subplot(2,2,2), imagesc(im_sing); title('image singel');

im_sing = imresize(im_sing, net.meta.inputSize(1:2)) ; % note: inputSize [32 32 3] subplot(2,2,3),imagesc(im_sing); title('resized image');

%% run the CNN res = vl_simplenn(net, im_sing) ;

% show the classification result scores = squeeze(gather(res(end).x)) ; [bestScore, best] = max(scores) ; subplot(2,2,4), imagesc(im) ; title(sprintf('%s (%d), score %.3f',... net.meta.classes.name{best}, best, bestScore)) ; `

[albanie]

You're almost there - you just need to remember to subtract the mean of the original training data before putting your new images through the network for classification.

Here's a script you can run in whichever of your directories contains the cifar data_batch_%d.mat files to compute the mean of the training data:

files = arrayfun(@(n) sprintf('data_batch_%d.mat', n), 1:5, 'UniformOutput', false) ;
data = cell(1, numel(files));

for fi = 1:numel(files)
  fd = load(files{fi}) ;
  data{fi} = permute(reshape(fd.data',32,32,3,[]),[2 1 3 4]) ;
end

data = single(cat(4, data{:}));
dataMean = mean(data, 4);

---

Finally, to use this in the example you gave above, insert the following line:

im_sing = im_sing - dataMean;

just before you call res = vl_simplenn(net, im_ing), and you should be good to go.

[Tomue]

Thank you very much! Now it is working, but why is dataMean needed, and why is it not calculated during the training process? I also had to change im2single(im) to single(im).

[albanie]

Before the network is trained, the dataMean is subtracted from all the input images so that they are centred around zero (this is done in the getCifarImdb function in cnn_cifar). This preprocessing step helps the network training process.

Once the network has been trained, any new images that you feed into the network need to be preprocessed in an identical manner to the training data - that's why it is necessary to subtract the data mean before classification.

[Tomue]

I trained a network with the help of the cifar example form the MatConvNet. After the training I tested the it with the following file. The result was unexpected for me. I tested 1000 different out of the 10000 imaged form the validation database a few times and I thought that about 80% should be categorized correctly but instead of 80% it just categorized about 20% correct. Could you please tell what's wrong? Tanks

My test file:

%% init
oldFolder = cd;
run setup;
cd(oldFolder);

% load the pre-trained CNN
net = load('cifar_cnn.mat');
net.layers(end)=[];

%% load and preprocess an image

load('cifar_batches.meta.mat')
load('cifar_test_batch.mat')
correct = 0; 
ncorrect = 0;

for i = 1:(length(data)/10)
fprintf('Image: %i\n',i)
im = permute(reshape(data(i,:)',32,32,3,[]),[2 1 3 4]) ;

figure(1)
subplot(2,3,1), imagesc(im);
title(sprintf('image to classify: %s',label_names{labels(i)+1}));

im_sing = single(im);
im_sing = imresize(im_sing, net.meta.inputSize(1:2)) ;  % note: inputSize [32 32 3]
im_sing = im_sing - net.dataMean;

% just for the Plot
im_sing2 = im2single(im); 
subplot(2,3,2), imagesc(im_sing2);
title('image im2single');

im_sing2 = single(im); 
subplot(2,3,3), imagesc(im_sing2);
title('image single');

im_sing2 = imresize(im_sing2, net.meta.inputSize(1:2)) ;  % note: inputSize [32 32 3]
subplot(2,3,4),imagesc(im_sing2);
title('resized image');

im_sing2 = im_sing2 - net.dataMean ;  % note: inputSize [32 32 3]
subplot(2,3,5),imagesc(im_sing2);
title('normalized image');

% run the CNN
res = vl_simplenn(net, im_sing) ;

% show the classification result
scores = squeeze(gather(res(end).x)) ;
[bestScore, best] = max(scores) ;
subplot(2,3,6), imagesc(im) ;
title(sprintf('%s (%d), score %.3f',...
net.meta.classes.name{best}, best, bestScore)) ;

if (best == (labels(i)+1))
    correct = correct + 1;
else
    ncorrect = ncorrect + 1;
end

end
%%
figure(2)
pie([correct,ncorrect],{'correct','not correct'});
title(sprintf('correct(%d) - not correct(%d)',correct,ncorrect)); 

[albanie]

If the network was trained using the default settings in the cnn_cifar example, contrast normalization and data whitening are performed alongside mean subtraction as a preprocessing step. This step needs to be replicated on new examples that are fed to the network.

Below is a modification of your code to perform this preprocessing on the raw test data (it produces identical data to the validation set members of your imdb file):

% load the pre-trained CNN and remove final layer
net = load('cifar_cnn.mat');
net.layers(end)=[];

% load test data
load('cifar_batches.meta.mat')
load('cifar_test_batch.mat')
correct = 0; 
ncorrect = 0;

% transform labels to index from 1
labels = labels + 1;

% Reshape data and perform mean subtraction
numImages = size(data,1);
data = single(data);
data = permute(reshape(data',32,32,3,[]),[2 1 3 4]) ;
data = bsxfun(@minus, data, net.dataMean);

% perform contrast normalization
z = reshape(data,[],numImages) ;
z = bsxfun(@minus, z, mean(z,1)) ;
n = std(z,0,1) ;
z = bsxfun(@times, z, mean(n) ./ max(n, 40)) ;
data = reshape(z, 32, 32, 3, []) ;

% whiten the data
z = reshape(data,[], numImages) ;
wData = load('wData.mat', 'W');
W = wData.W;
[V,D] = eig(W) ;
d2 = diag(D) ;
en = sqrt(mean(d2)) ;
z = V*diag(en./max(sqrt(d2), 10))*V'*z ;
data = reshape(z, 32, 32, 3, []) ;

for i = 1:(length(data)/10)
   fprintf('Image: %i\n',i)
    im = data(:,:,:,i);
    res = vl_simplenn(net, im) ;

   % show the classification result
    scores = squeeze(gather(res(end).x)) ;
    [bestScore, best] = max(scores) ;

    if (best == labels(i))
        correct = correct + 1;
    else
        ncorrect = ncorrect + 1;
    end

end

NOTE: The variable W must be recomputed from the training data before this code is run. This can be done by adding the line save('wData.mat', 'W') into the getCifarImdb function call in cnn_cifar when generating the imdb file and reloading it as shown here.

[Tomue]

Thank you so much @albanie, now that is working fin!

But actually i have a new problem with the result score for the cnn or with the interpretation of this score. --> #520 Can anybody help me?

Thanks a lot!