Check failed: background_label_id != label (0 vs. 0) Found background label in the dataset

[Bardo91]

Hi, I am trying to train the network. I am following the steps given in "https://github.com/weiliu89/caffe/wiki/Train-SSD-on-custom-dataset" I created the lmdb files for training and testing. Everything seems ok but when it started the train, while checking the bboxes (in file bbox_util.cpp) it crashes because of the error in the title. I get the following output:

I0523 17:01:31.303931 16432 solver.cpp:75] Solver scaffolding done.
I0523 17:01:31.305892 16432 caffe.cpp:251] Starting Optimization
I0523 17:01:31.305915 16432 solver.cpp:294] Solving VGG_VOC0712_SSD_300x300_train
I0523 17:01:31.305919 16432 solver.cpp:295] Learning Rate Policy: multistep
I0523 17:01:31.313360 16432 blocking_queue.cpp:50] Data layer prefetch queue empty
F0523 17:01:35.889597 16432 bbox_util.cpp:1066] Check failed: background_label_id != label (0 vs. 0) Found background label in the dataset.
*** Check failure stack trace: ***
    @     0x7f00cfcef5cd  google::LogMessage::Fail()
    @     0x7f00cfcf1433  google::LogMessage::SendToLog()
    @     0x7f00cfcef15b  google::LogMessage::Flush()
    @     0x7f00cfcf1e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f00d057ed71  caffe::GetGroundTruth<>()
    @     0x7f00d03f5f7e  caffe::MultiBoxLossLayer<>::Forward_cpu()
    @     0x7f00d0541537  caffe::Net<>::ForwardFromTo()
    @     0x7f00d05418a7  caffe::Net<>::Forward()
    @     0x7f00d05db510  caffe::Solver<>::Step()
    @     0x7f00d05dbf9e  caffe::Solver<>::Solve()
    @           0x40bcf4  train()
    @           0x4077c8  main
    @     0x7f00ce486830  __libc_start_main
    @           0x408099  _start
    @              (nil)  (unknown)
Aborted (core dumped)

Does anyone know the reason?

[Bardo91]

I end-up adding a label called background and in the python file I set the label ID to that label

[MyVanitar]

@Bardo91

I have background-only images like you in the dataset and I don't know how should I feed them to SSD. may I ask you explain a bit more?

[Bardo91]

@VanitarNordic I hadn't exactly that problem when I put this issue. What it happened to me is that I forgot to add a label for the background in the configuration files.

Regarding to your problem.... How are you training the net, using the pascal_voc structure with xmls? And how are you generating your custom xmls?

[Bardo91]

I say that you asked about it in https://github.com/weiliu89/caffe/issues/715. It seems that you are not the only one facing this issue. In issues https://github.com/weiliu89/caffe/issues/146 and https://github.com/weiliu89/caffe/issues/348 they asked the same. Wei Liu ended up saying that the key is to modify a method called "MineHardNegatives". @abhisheksgumadi asked about it and seems that digged into the code (mentioning the line where the method is defined in Feb 2017). However, I cant find any commit related with it on his forked repository (https://github.com/abhisheksgumadi/caffe/tree/ssd), actually the last one is on Nov2016.

[MyVanitar]

Actually I removed the <object> </object> section inside background-only images' xml annotation. The model trains, but I am not quite sure if it trains in a correct manner.

[MyVanitar]

Actually I have a dataset and I want to test which object detection model handles a better accuracy. This dataset contains some background-only images also.

The YOLO trained with it and I have the results. I did a fine-tuning with SSD in low number of iterations. it showed between 85% to 86%mAP, but the detection results are not that much good (in comparison with YOLO). it can not detect small objects and also some false positives. YOLO did not show these. Therefore I have a doubt if it trains correctly.

[MyVanitar]

Here is the screenshot. I'm not quite sure but maybe the loss and mbox_loss are high.

[Bardo91]

Well, both YOLO and SSD are very good solutions, YOLO has new updates in December 2016 (https://arxiv.org/abs/1612.08242) and SSD is from Dec 2015 (https://arxiv.org/abs/1512.02325). Reading the statistics in the articles they are supposed to have (approx) similar results using the same resolution. Actually, in the latest article of YOLO (YOLOv2) it sais that SSD512 has better mAP than YOLO, but YOLO focuses on improving the speed.

Regarding to the comments that Wei Liu said in the issue https://github.com/weiliu89/caffe/issues/146, it looks like training with background-only images is not that simple. I gave a quick look to the method and seems like the background predictions are ignored (https://github.com/weiliu89/caffe/blob/9d6e8151eedf3e8d3abaecde47b788a1ec2d2156/src/caffe/util/bbox_util.cpp#L765 ; https://github.com/weiliu89/caffe/blob/9d6e8151eedf3e8d3abaecde47b788a1ec2d2156/src/caffe/util/bbox_util.cpp#L798). For futher information a deeper research might be needed. I think that, at this point, YOLO is having more information, and the SSD is just doing nothing for those background-only images. Maybe @weiliu89 can confirm it.

[MyVanitar]

@Bardo91 @weiliu89

actually I've asked him in a separate issue also, and waiting for his reply.

[IsaacRe]

@Bardo91 Those two lines of code are only reached when 'share_location' is set to False, which shouldn't ever be the case, as @weiliu89 mentioned in these threads: #473, #275

[adithya-p]

@Bardo91 I'm having the same issue but adding a label called background and setting the label id to that label in the python file didn't help.

The error still persists. I am doing exactly what is specified on the wiki to train on custom dataset. I have around 60k images with background as also a label to eliminate false positives. (Please note that I don't have any image with only background.) Can someone shed some light on this issue?

CC: @weiliu89 @fraukej @nyanmn