[object detection] support for per-example partial labels

[staubda]

Prerequisites

Please answer the following question for yourself before submitting an issue.

• [X] I checked to make sure that this feature has not been requested already.

1. The entire URL of the file you are using

The following branch contains my changes to enable support for this feature: https://github.com/staubda/models/tree/object_detection_partial_data

Files affected include:

• https://github.com/tensorflow/models/blob/master/research/object_detection/core/losses.py
• https://github.com/tensorflow/models/blob/master/research/object_detection/data_decoders/tf_example_decoder.py
• https://github.com/tensorflow/models/blob/master/research/object_detection/inputs.py
• https://github.com/tensorflow/models/blob/master/research/object_detection/meta_architectures/ssd_meta_arch.py

2. Describe the feature you request

I'd like to contribute a PR adding per-example support for partial labels to the object detection repo. Each example image should be (optionally) bundled with a list of classes for which labels were annotated, which may include any subset of the complete taxonomy supported by the model. During training each example should only contribute loss for the classes it supports.

This is particularly relevant when combining datasets with images from the same domain labeled with partially overlapping or non-overlapping taxonomies, with the goal of training a model supporting a union of the taxonomies. E.g. dataset1 is annotated with labels for {car, motorcycle, truck}, dataset2 is annotated with labels for {motorcycle, pedestrian, bicycle}, and the detection model should support {car, motorcycle, truck, pedestrian, bicycle}. In order to train the best model, predictions of, e.g., bicycle, on an example from dataset1 should not be penalized since a bicycle may actually be present but will never be labeled.

3. Additional context

Primary changes are to

• Loss function
• Detection meta-arch
• TFRecord deserialization

I've tested the feature on a synthetic dataset mimicking the above scenario (a subset of COCO images with labels partially removed) and recorded the results in this repo https://github.com/staubda/partial-data/tree/master/object_detection/trained_models. Specifically, runs 4, 5, and 6 show results on an identical test and train sets, but with:

• run 4: partial labels, no per-example support for specifying and masking unsupported classes when computing loss - mAP@0.5 = 0.196
• run 5: complete labels - mAP@0.5 = .251
• run 6: partial labels, includes per-example support for specifying and masking unsupported classes when computing loss - mAP@0.5 = 0.201

The repo includes all notebooks and scripts necessary for recreating datasets and training runs.

4. Are you willing to contribute it? (Yes or No)

Yes, PR is ready to go for SSD meta-arch with weighted sigmoid classification loss.

[sayakpaul]

This seems very interesting and useful. So, if I am understanding correctly, this PR would allow us to do multi-label classification along with the regular detection? Or would be able to add a quick pictorial depiction of the concept?

[staubda]

Not adding any additional classifiers to the model. Essentially the idea is to enable the user to specify that for a given image, only a certain subset of the object classes were labeled; therefore during training, don't penalize the model for predicting object classes outside of that subset.

So, for example, let's say you find a dataset of highway images where cars are boxed (dataset1), and a second highway dataset where only trucks are boxed (dataset2), and you'd like to train a single model that can predict boxes for both cars and trucks. There are probably trucks in dataset1 and cars in dataset2, however they won't be labeled. If you combine the two datasets into a single dataset and train the model, it will (correctly) start detecting trucks in dataset1 and cars in dataset2 during training, but each time it does it will be penalized because those objects weren't labeled. So this feature just lets you say that, for a given image, cars (e.g.) were not labeled, so don't penalize the model for predicting car because it might actually be correct.

This paper discusses training on partial labels. It's in the context of image classification rather than object detection, but the idea and motivation are the same.

[sayakpaul]

Ah, now I understand. Thank you.

[staubda]

@pkulzc any interest in adding this feature? seemed like you had already started laying some of the groundwork.

[alexdwu13]

Any updates? I'd love to use this feature asap.