search

NVIDIA / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
672 stars 263 forks source link

resnet50-ssd training issues #548

Closed kmakonyi closed 5 years ago

kmakonyi commented 5 years ago

Hello,

I tried to train resnet50 with ssd on my own dataset. The model comes from https://github.com/jay-mahadeokar/pynetbuilder/tree/master/models/voc2007_ssd/resnet_50 the pretrained weights from https://www.dropbox.com/s/en1niufn7nagh27/detect_snapshot_iter_60000.caffemodel?dl=0. I used this command to train: ../caffe_nvidia_0.17.1/build/tools/caffe train --solver="solver.prototxt" --weights="detect_snapshot_iter_60000.caffemodel" --gpu 0 2>&1 | tee jResnet50_440x440.log I am trying to train on the open imagege dataset (https://storage.googleapis.com/openimages/web/index.html)

As a result I got constant 0 loss. It looks that the source of the issue is that the MineHarExamples function in the multibox_loss_layer function always gives 0 to the nummaches. (I got the same issue if I modify the provided ssd_pascal.py script in order to train resnet50. Using the original ssd_pascal.py (with VGG) I cant get even this far, because I got error during the model creation: Check failed: priorbox_param.variance(i) > 0 (0 vs. 0))

Can you see any obvious error that I made here?

Thanks for the investigations in advance. Regards, Karoly

drnikolaev commented 5 years ago

Hi @kmakonyi I'd start from verifying that pre-trained model has identical labeling with the open images dataset.

kmakonyi commented 5 years ago

Hello, Thank You for your answer. I don't really know what do you mean? I try to fine-tune the model with only two classes. I created the following labelmap: item { name: "none_of_the_above" label: 0 display_name: "background" } item { name: "Human" label: 1 display_name: "Human" } item { name: "Vehicle" label: 2 display_name: "Vehicle" } The pre-trained model was trained on voc2012, so the labeling is obviously not the same. ... but, is this the problem? What is your suggestion to solve this problem? The vehicle class merges the bus, bike, car, motorbike classes that are individual classes in the voc ... how to merge them into one class?

Thanks for your help. Regards, Karoly

drnikolaev commented 5 years ago

This is the problem actually. If voc2012 has labels 0, 1, 2 assigned to something else, you should choose other values. As of merging, I'm not sure I can help here.