Trained model detects other objects

[diashima]

Hello! I have trained my model using /object_detection/model_main.py with SSD_mobilenet_v1_coco model. I have trained it to detect glasses for something like ~20,000 steps, after the training it detects glasses with 99-100% accuracy. But it also detects everything, that usual mobilenet v1 coco model detect(around 90 other objects) and labels them as glasses. What i must do to make my model stop detecting other objects?

System information

• What is the top-level directory of the model you are using: trained model
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): tensorflow-gpu
• Bazel version (if compiling from source): n/a
• CUDA/cuDNN version: 10.1
• GPU model and memory: 2080ti 11264 MB
• Exact command to reproduce: n/a

[Vinod-Koli]

Before training tensorflow models on your data you should update num_classes:, max_detections_per_class:, max_total_detections: etc.. in pipeline.config and

create your lable_map.pbtxt, for example

item {
  id: 1
  name: 'Wine Glass'
}

item {
  id: 2
  name: 'Cup'
}

did you update these files ?

[diashima]

@Vinod-Koli, i have edited num_classes to 1 and created my own label map: item { id: 1 name: 'glasses' }

Do you think editing max_total_detections and max_detections_per_class might help? Will try later

[diashima]

Here is an example, it predicts well glasses after my training, meanwhile same model can detect keyboard(as it was in 90 COCO pretrained dataset)

[netanel-s]

It is a common practice to have 1:3 positive-to-negative examples to the detector. I guess that you didn't provide negative examples to your detector, only glasses positive examples. Since you're using a pre-trained model which was trained to detect COCO objects, it has a prior to detect these objects. If you're not giving the detector negative examples for these objects, he probably won't "forget" them very easily. It might forget them e.g. if you're training on a lot of other classes, regularize it harshly, etc. Bottom line: take images from COCO, and use them as negative examples for objects which are not glasses.

[diashima]

@netanel-s, using as negative example means label other objects, create xml, include them to .csv and .record files and then train the model to detect "glasses" and "anything that not glasses"? How i must label negative images?

Sorry for asking stupid questions, literally can't understand xD

[Vinod-Koli]

It is a common practice to have 1:3 positive-to-negative examples to the detector. I guess that you didn't provide negative examples to your detector, only glasses positive examples. Since you're using a pre-trained model which was trained to detect COCO objects, it has a prior to detect these objects. If you're not giving the detector negative examples for these objects, he probably won't "forget" them very easily. It might forget them e.g. if you're training on a lot of other classes, regularize it harshly, etc. Bottom line: take images from COCO, and use them as negative examples for objects which are not glasses.

@netanel-s Referring to this post seems like negative images are not needed as we are providing bounding boxes, anything outside box is considered as negative

but if you want to provide negative images you should have one more class itself for 'None'!!

[netanel-s]

@Vinod-Koli , that's not exactly right. You're right that every anchor that doesn't match a ground truth bounding box is trained as negative. But this means that your negatives are only from images you're giving to the model. For example, if you train on glasses, than the entire person wearing the glasses will be a negative. But what about keyboard? If the images he used in the training set doesn't include keyboards, then the model never seen a keyboard as a negative example. @diashima , you can simply add the images to your training set, without ground truth boxes. Since these images don't have any ground truth boxes, everything in it will be considered as negative.

[diashima]

@netanel-s, you mean that images must be inside images folder, path to which i use when i create train.record? without having any information about that images in .csv file?

[netanel-s]

@diashima sort of. When you convert your training images to the train TF record, include also images which will be used as negative examples. I didn't say you don't need to include their meta data, but rather that they don't need to include any bounding boxes, since they're negative examples rather than positive.

[diashima]

@netanel-s, thank you a lot! will try it later

P.S. actually my .csv will be something like? glasses.jpg,251,201,glasses,31,77,225,139 glasses2.jpg,213,237,glasses,43,98,128,132 glasses3.jpg,243,207,glasses,79,61,152,81 nonglasses.jpg,XXX,YYY nonglasses2.jpg,XXX,YYY

[Vinod-Koli]

@Vinod-Koli , that's not exactly right. You're right that every anchor that doesn't match a ground truth bounding box is trained as negative. But this means that your negatives are only from images you're giving to the model. For example, if you train on glasses, than the entire person wearing the glasses will be a negative. But what about keyboard? If the images he used in the training set doesn't include keyboards, then the model never seen a keyboard as a negative example. @diashima , you can simply add the images to your training set, without ground truth boxes. Since these images don't have any ground truth boxes, everything in it will be considered as negative.

@netanel-s Yes that's right! But in my opinion i think this issue is not because of no negative images, It would not detect other objects with 99% accuracy! So I feel the problem is due to something else..!

[diashima]

@Vinod-Koli, by the way it detects other objects only from COCO dataset, so i think it is something connected with "old" memory

[wander1985]

Maybe by removing or commenting the fine_tune_checkpoint: in your .config file would work.

[SiddharthSingi]

I am facing a similar issue, I have trained my custom dataset (which only consists of 12 classes like car, bus, bicycle, pedestrian etc.) on mobilenet_ssd_v2 which was pre trained on coco dataset. Two problems I am facing are:

1) My model has previous memory and also detects objects like bottles, and keyboard, labelled with my new classes

2) When I test my model, I can see that the model is predicting a total of 100 classes!!! There are a lot of extra classes, boxes and predictions being made apart from my dataset classes that are being predicted. @diashima could please check if you are also facing a similar issue. These extra predictions are resulting in an increased time lag of my inference.

Also can someone please tell me how would fine_tune_checkpoint affect my training

[netanel-s]

It is not recommended to comment out fine_tune_checkpoint, since then you'd train the entire model from scratch (assuming there isn't a checkpoint in model_dir, otherwise it'd load it and fine tune it). What you can do is one of the following:

• In case you have the flag load_all_detection_checkpoint_vars, then comment it out or set it to false. It will initialize more detection variables, hopefully helping you forget the other classes.
• Set the fine_tune_checkpoint_type to "classification" rather than "detection". The effect should be similar.
• If both don't solve it, you can fine-tune from the backbone-only checkpoint (e.g. mobilenet_v1_1.0_224.ckpt) rather than a checkpoint fine-tuned on COCO (SSD_mobilenet_v1_coco).

[diashima]

@netanel-s, can you please help me about my .csv file's content. Must it contain image.jpg,Xsize,Ysize lines of negative image examples?

Even if I add these lines, i can't convert .csv to .record using defaultgenerate_tfrecord.py

[Vinod-Koli]

@netanel-s, can you please help me about my .csv file's content. Must it contain image.jpg,Xsize,Ysize lines of negative image examples?

Even if I add these lines, i can't convert .csv to .record using defaultgenerate_tfrecord.py

@diashima I was wondering if the issue is resolved ?

[diashima]

@Vinod-Koli, I didn't add absolutely negative examples, because I don't know how to create proper .record with negative examples. I added one more class, made more training steps and more images. So other objects are rarely detected. But i guess if I had known they way of creating .record with no bounding boxes images the model would have been more precise

[Vinod-Koli]

@Vinod-Koli, I didn't add absolutely negative examples, because I don't know how to create proper .record with negative examples. I added one more class, made more training steps and more images. So other objects are rarely detected. But i guess if I had known they way of creating .record with no bounding boxes images the model would have been more precise

@diashima @netanel-s In my case it did not detect any objects from original classes I checked this behavior, previously i had trained tensorflow object detection model on a very small dataset of traffic lights (just 50 images), when ran the model on the images from original dataset, it did not detect! (detection probability is below 5%), works as expected.

[AshishGusain17]

@Vinod-Koli, I didn't add absolutely negative examples, because I don't know how to create proper .record with negative examples. I added one more class, made more training steps and more images. So other objects are rarely detected. But i guess if I had known they way of creating .record with no bounding boxes images the model would have been more precise

Have u got the answer. I am stuck in a similar problem. I have one class to detect. How to add negative images??

if row_label == 'closed':
    return 1
else:
    return 0

Right now, I am adding negative images with a different label and bounding boxes as 0,0,0,0 in the csv file. Is it correct?

[jenapss]

@diashima @AshishGusain17 @Vinod-Koli Guys, who could find workarounf to creating tf_records and csv file with negative images? I found detailed explanation here https://github.com/tensorflow/models/issues/3365#issuecomment-661326737 but still the code snippet he gave doesn't work for creating csv files with negative images. Did you find a solution to this problem? How to create a tf_record with negative images? Thanks in advance!

[AshishGusain17]

@jenapss I read somewhere that tensorflow itself takes negative images from those outside of the bounding boxes.

[jenapss]

Thank you for your reply! So you mean we should add just more positive images that have bboxes?

[mahamatnoumai]

@jenapss I've encountered similar problem while training tensorflow object detection using sdd mobilent. I would like to know if you were able to fix the issues, and manage to added background images. and also does help to increase the detection accuracy? Thanks.