Closed BeSlower closed 8 years ago
rksltnl
commented
8 years ago Are you finetuning from a pretrained model on your dataset? If so, you should do the mean image subtraction in the data layer. Look at step 2 in the "training procedure" section.
Download the ILSVRC12 ImageNet mean file for mean subtraction. Refer to Caffe the ImageNet examples here.
Also, try increasing your batch size to 128.
BeSlower
commented
8 years ago Yes. I try to fine tuning from a model trained from inception v3. I do subtract the mean value. you can see the data layer in my log.
layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
transform_param {
mirror: true
crop_size: 299
mean_value: 103.94
mean_value: 116.78
mean_value: 123.68
}
image_data_param {
source: "/home/hikvision/shizhan/Deep-Metric-Learning-CVPR16-master/train.txt"
batch_size: 32
new_height: 320
new_width: 320
}
}If set the batch size = 128 directly, it will cause "our of memory" error. Therefore, I set the iter_size = 4 and batch size = 32, which is same with batch size = 128.
rksltnl
commented
8 years ago This isn't consistent with the mean image subtraction with imagenet_mean.binaryproto explained in the readme.
crop_size: 299 mean_value: 103.94 mean_value: 116.78 mean_value: 123.68
Also, try lowering your learning rate.
Plus your question has nothing to do with this repository. Ask https://groups.google.com/forum/#!forum/caffe-users for caffe related help.
BeSlower
commented
8 years ago Thanks a lot. But I use my own dataset and a pretrained model from inception v3. I just use your loss layer. Is it necessary that the mean value should be equal to the mean value of Imagenet dataset? Even though I fine tune the model on my own dataset.
Anyway, thanks.
rksltnl
commented
8 years ago It depends. Which dataset was it pretrained on? If you're using https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet, you should use the imagenet mean file. Have you tried lowering your learning rate? Once again, this is a question about Caffe not about this repository. Ask https://groups.google.com/forum/#!forum/caffe-users on how to use Caffe not here.
BeSlower
commented
8 years ago Thanks for your time.
JoinWei
commented
7 years ago Hello,do you solve the problem. I meet the same problem as you. Whatever I set the learning rate, the loss is nan. I also just use the loss layer in my model, subtract mean values as yours. If you have solved the problem, please tell me the solution. Thanks!
Hi, At first time, when I set the learning rate = 0.01, the loss shows nan for all kinds of embedding size. Therefore, I decrease the learning rate to 0.0001. It works for all embedding size except 64. Could you help me to check it? Attached is the log when embedding size = 64.
Another thing is that I did't follow your procedure to generate the lmdb data because you don't have too many documentations about the input organization or format for your Matlab code. So, I use the image data layer at the beginning of the network. I just wanna make sure it won't conflict with the loss layer you implemented.
(update) Finally, loss = nan for all kinds of embedding size.
Best regards, Zhan