How to retrain existing SSD_Mobilenet_MSCOCO model for a new class label

[humayun]

Dear all,

I run the trained model (ssd_mobilenet_v1_coco_11_06_2017) to detect bounding box on my dataset, but this trained model is not performing well because the main object that I am detecting is not in trained model. Here is example that I am following: https://github.com/tensorflow/models/blob/master/object_detection/object_detection_tutorial.ipynb

First Question: How can I retrain the existing ssd_mobilenet_v1_coco_11_06_2017 model with one addition class. For the new class, I generate boundingbox annotation in xml format.

Second Question: How can I train a new model only with my own dataset which has only two classes. If possible share any demo or code or tutorial.

Thank you.

-Humayun

[derekjchow]

There currently is not way to just add one more class to a pre-existing model. Your best bet would be to train a new model using our Mobilenet checkpoint for finetuning.

To answer your second question, we've just added documentation about using your own data. This requires that you do have a dataset (images + box labels) in a format you're familiar with and will walk through converting the data set for TF Records.

[saikishor]

@humayun Did you find any possible way to add new class to the pre-trained model?. If yes, please let us know.

[realwol]

@saikishor http://androidkt.com/train-object-detection/ https://pythonprogramming.net/testing-custom-object-detector-tensorflow-object-detection-api-tutorial/?completed=/training-custom-objects-tensorflow-object-detection-api-tutorial/

[saikishor]

@realwol Thanks for the comment. Using the links you posted, we will be able to design a new model for the respective classes it is trained on, but I would like to know, how to add additional classes to the model. For instance, models trained on COCO dataset have 90 classes, I would like to know, how to add one more class to that model and get a model with 91 classes as an outcome of it. I guess this is what @humayun is interested.

[realwol]

@saikishor https://www.tensorflow.org/tutorials/image_retraining#other_model_architectures I think this is what you want.

[odilialirani]

@saikishor hi, I'm just wondering if you've found anything about adding a class to the model?

[dfoley84-zz]

I Downloaded the COCO data-set along with the Annotations but unsure how to run it. any one know how to run this ?

[ryanphung]

@dfoley84 you'll have to first convert the COCO data-set into TF Record, using a script like this: https://github.com/offbye/tensorflow_object_detection_create_coco_tfrecord

[LaurentBerder]

@ryanphung I've downloaded the COCO dataset and converted it to a TF Record. I've also labeled my own custom dataset and converted it to a TF Record.

What I'd like to know is if there's a way to combine both TF Records to train a model on the addition of COCO + Custom classes...

[cantren]

Is there something specific about this model that is preventing you from using Transfer learning? If you can recover the weight matrices from the pretrained model, you should be able to freeze all the weights except for the final fully connected layer, or am I missing something?

On Wed, Mar 7, 2018 at 7:21 AM, Laurent Berder notifications@github.com wrote:

@ryanphung https://github.com/ryanphung I've downloaded the COCO dataset and converted it to a TF Record. I've also labeled my own custom dataset and converted it to a TF Record.

What I'd like to know is if there's a way to combine both TF Records to train a model on the addition of COCO + Custom classes...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tensorflow/models/issues/1809#issuecomment-371171604, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZN_iiiyX0orgd_niqhZSzCypb0jtQsks5tb_opgaJpZM4OI90H .

[LaurentBerder]

@humayun, I'm trying with a new approach: Instead of combining TF Records together, I'm doing it before converting both my dataset and the COCO dataset from JSON to TF Records.

I made a python program to do so, see below. However, it is of course highly customized to the format I had which is not so common...

import json
import os
import argparse
from PIL import Image

def add_info_to_coco_json(args):
    with open(args.COCO_annotation_file, 'r') as f:
        coco = json.load(f)

    # Check the highest id numbers for images and annotations, we'll work incrementally from there
    im_ids = max([im['id'] for im in coco['images']])
    an_ids = max([im['id'] for im in coco['annotations']])

    # Add info to 'categories'
    cats = {'new_category_1': 91, 'existing_category_1': 8, 'new_category_2': 92}
    coco['categories'].append({'id': 91, 'name': 'new_category_1', 'supercategory': 'outdoor'})
    # coco['categories'].append({'id': 8, 'name': 'existing_category_1', 'supercategory': 'vehicle'})
    coco['categories'].append({'id': 92, 'name': 'new_category_2', 'supercategory': 'vehicle'})
    )

    # Loop over custom files
    for file in os.listdir(args.custom_annotation_folder):
        if '__labels.json' in file:
            with open(args.custom_annotation_folder + file, 'r') as f:
                img = json.load(f)
            im_ids += 1

            filename = img['image_filename']
            print(filename)
            with Image.open(args.custom_annotation_folder + filename) as im_file:
                width, height = im_file.size  # Image height and width

            # Add info to 'images'
            new_image = {'id': im_ids, 'filename': filename, 'width': width, 'height': height}
            coco['images'].append(new_image)

            # Add info to 'annotations'
            for label in img['labels']:
                an_ids += 1
                xmin = (label['centre']['x'] - (label['size']['x'] / 2)) / width
                ymin = (label['centre']['y'] - (label['size']['y'] / 2)) / height
                new_annotation = {'id': an_ids, 'category_id': cats.get(label['label_class']),
                                  'bbox': [xmin, ymin, width, height], 'image_id': im_ids}
                coco['annotations'].append(new_annotation)

    return(coco)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--COCO_annotation_file', help='The COCO annotation file downloaded from COCO website')
    parser.add_argument('--custom_annotation_folder', help='The folder in which json files are saved')
    parser.add_argument('--output_json', help='The new combined annotation file to save')
    args = parser.parse_args()
    new_coco = add_info_to_coco_json(args)
    with open(args.output_json, 'w') as output:
        json.dump(new_coco, output)

if __name__ == '__main__':
    main()

[dfoley84-zz]

I have COCO Training Images and Wider Face I don't have a GPU so I would have to retrain on a CPU,

What I am interested in is a detection of a Person along with a Face detection, does anyone know if there is a pre-trained model on this already?

[cirovarriale]

@LaurentBerder That method worked? If I understand something, in your script you did the following operations:

Starting from the COCO annotation file that should be a json containing:

a. categories: object to recognize (id, name and supercategory=?) b. images: image dataset used for training (id, filename, width, height) c. annotations: info about the labeling operation (id, _categoryid=?, bbox, image_id)

• Add 2 categories (new_category_1 and new_category_2) to coco json file with coco['categories'].append
• Add some images to coco images coco['images'].append
• Add some annotations to coco annotations coco['annotations'].append

and return the new COCO annotation file.

So summing up: you modified the coco annotation file in such a way to insert two new classes. To do this you added manually the two classes in the categories voice. While for all annotations file __label.json, you take info about the image and put them into images voice, and info about the bounding boxes and put them into the annotation.

Didn't understand this: cats.get(label['label_class']) (cause there is no label_class voice in cats).

So I want to know:

• this method worked?
• if I have a trained model that recognize only 1 class. And then I want to add another 1 class to recognize, can be enought to do somethingh like this:
  1. put togheter all images (old and new) and all xml files (old and new)
  2. create now the new TFrecord file
  3. modify the config file with the new 'num_classes=2'

This method could worked also for you I think. Or is it wrong?

[cirovarriale]

@LaurentBerder That method worked? If I understand something, in your script you did the following operations:

Starting from the COCO annotation file that should be a json containing:

a. categories: object to recognize (id, name and supercategory=?) b. images: image dataset used for training (id, filename, width, height) c. annotations: info about the labeling operation (id, _categoryid=?, bbox, image_id)

• Add 2 categories (new_category_1 and new_category_2) to coco json file with coco['categories'].append
• Add some images to coco images coco['images'].append
• Add some annotations to coco annotations coco['annotations'].append

and return the new COCO annotation file.

So summing up: you modified the coco annotation file in such a way to insert two new classes. To do this you added manually the two classes in the categories voice. While for all annotations file __label.json, you take info about the image and put them into images voice, and info about the bounding boxes and put them into the annotation.

Didn't understand this: cats.get(label['label_class']) (cause there is no label_class voice in cats).

So I want to know:

• this method worked?
• if I have a trained model that recognize only 1 class. And then I want to add another 1 class to recognize, can be enought to do somethingh like this:
  1. put togheter all images (old and new) and all xml files (old and new)
  2. create now the new TFrecord file
  3. modify the config file with the new 'num_classes=2'

This method could worked also for you I think. Or is it wrong?

[LaurentBerder]

@cirovarriale Yes, my method worked.

Your explanation is correct. I originally had one json file per labelled image (with a different way of recording bbox coordinates from TF), instead of a single large file for all training data.

Eventually, I ended up dropping the use of TensorFlow (too cumbersome for little added value) and used a script to convert the other way around (from COCO to individual json files) which you can find in my repository if you're interested.

[SriramganeshMarimuthu]

I would like to know, If it is possible can we retrain the object detection model?

Say Example: I have trained my own model to identify red and green light. Now I need to update/retrain my model to identify the yellow color light.

When we use TF- Image Classifier, we can add new image sets to test and train directory and we can execute retrain.py to update the .pb file to get new results.

Could you please guide me, how to implement retrain mechanism for object detection model with TF?

[kirubade]

Is anybody able to achieve training custom object in-addition to existing 90 coco pre-trained objects?

[LaurentBerder]

@kirubade I was, but not in TensorFlow. I had to revert to Chainer CV as it was more practicable.

[SaurabhC1010]

Is there a way to use pretrained OID models for retraining as coco is not performing well for custom objects like berries.

[Rahul-Aanand]

@LaurentBerder @kirubade Any update on retraining a pre-trained model to detect objects of new classes in addition to preserving object detection for previously trained classes (detecting 90+1 classes) using Tensorflow?

@LaurentBerder Any document explaining step by step process to follow what you have done in your repository?

[LaurentBerder]

@Rahul-Aanand As I explained earlier, I have tried for a while the retraining, but didn't manage to do it in Tensorflow, so I ended up changing to ChainerCV because it was easier to handle.

Concerning my repository, the file I gave a link to (from_COCO_to_custom_json.py) translates the labels to the same format that the one used by the manual labeling tool, and copy the concerned image files to the same folder. This uses the label YAML file to only select the files containing the type of objects that you want to use for training.

I'd recommend you read the README to the general project, with better description, and if you need more info about it, you should ask them directly to me on the repository rather than here on Tensorflow's.

[a428tm]

@LaurentBerder Thank you for sharing the project link.

I do understand that you used ChainerCV and not TF, but because I have to use Coral, I wanted to ask you some Qs about your method in regards to TF. Also, @cirovarriale kinda summarized it, but I just wanted to clarify if you dont mind.

Your repo, from_COCO_to_custom_json.py, it reads from COCO annotation. Is the annotation file from here? - cocodataset.org

So you basically took the file captions_train2017.json and added your file and annotation. For example - ..... "images": [{"license": 4,"file_name": "000000397133.jpg","coco_url": "http://images.cocodataset.org/val2017/000000397133.jpg","height": 427,"width": 640,"date_captured": "2013-11-14 17:02:52","flickr_url": "http://farm7.staticflickr.com/6116/6255196340_da26cf2c9e_z.jpg","id": 397133},{.... (Although this is from captions_val2017.json) You just appended it to this list, correct

Did you, by any chance, download 2017 dataset? I tried opening captions_train2017.json but my fan basically started going off. So I couldn't really even take a look at what was inside. But I wanted to see if that is what you appended to.

You didn't use TF not because the workflow is any different, correct? I guess what I am asking is, based on your experience and knowledge, do you foresee any issue if I went and did the same with TF?

Thanks, Jae

[aafaqin]

How to train for a class of images downloaded from OID (open image dataset) specifically for object detection. I downloaded the CSV file for annotations but the CSV file is very large almost 2.3Gb and that file includes all the annotations which are not even in my dataset. How do I create a TFrecord for the same.