Add ignore regions to reduce effect from synonymous objects

[Trouble404]

some datasets like CrowdHuman contains 'ignore regions' bounding boxes annotation, there are 100k ignore bounding boxes in CrowdHuman, when we train a detector based on this dataset, we may want to reduce the effect from human doll, human billboard or dense crowd human. these part of regions should not consider as positive and negative supervise, bug I notice Detectron2 don't have a feature to process 'ignore' annotation during dataloader and matcher, this feature cound be used for lots of projects which need ignore labels

[ppwwyyxx]

It's unlikely we'll add to existing models large features that are specifically tailored to properties of certain datasets. Such features are expected to be implemented separately as new modules in users' code, or potentially in detectron2/projects if they appear more popular and widely useful.

If there are any good utilities (e.g., IOU vs IOA) or more general improvements that can simplify the above process or are part of the abovementioned feature, they can be discussed separately as individual feature requests.

[ppwwyyxx]

I just noticed https://github.com/facebookresearch/detectron2/pull/1910 and it gave a better idea of what's needed to implement this. For reasons above (and more in docs) we may not be able to accept it in its current form, but some independent pieces of it can be polished and included. For example, if we for some reason need to implement this model, we'll take the following approach:

1. start by implementing a mechanism to pump the extra data into the model

   • allow Intances to store custom image-level data, if it's easier to use than storing "ignore" as a per-instance annotation. after discussions, we'll not change the semantic of Instances but just pass them as extra arguments instead

2. (in parallel with 1) implement necessary utility functions such as IoA (or "iof")

3. implement the ignore strategy for one model (e.g. RPN), and potentially more afterwards

Whether (3) can go into the existing models, or should be implemented separately outside detectron2 will depend on its complexity, which also relies on work done in (1) and (2). If you like to add them, they can be done in separate PRs (or issues first, if they involve significant changes). Implementing many orthogonal changes for a non-project directory in a single large commit is less effective and something we usually don't do.

[Trouble404]

Thanks for your replying and suggestions, I will try to split those features and re-implement them.

[Trouble404]

I have re-implemented those three features, and pull-request the first two features. based on 'pump instance' and 'ioa', the ignore strategy for RPN is efficient to implement without large modify this time.

[ppwwyyxx]

We can proceed as follows:

Let's use an extra Instances object to represent ignored regions. In this particular dataset this object perhaps only contains ignored_boxes or boxes field, but it can potentially contain more. This object should be in model's batched_inputs as an extra key. There are the following components that we can implement in projects/IgnoredRegions to support this.

1. Necessary data loading code to produce this format. This consists of two parts:

   • Parsing the dataset. This probably means calling load_coco_json with extra_annotation_keys=["ignore"]. Note that we probably can't add "ignore" into the load_coco_json function because IIRC this is not part of the COCO specification.
   • It would be ideal to add support for crowdhuman dataset directly into detectron2.
   • Turning the loaded data into model's input format. This is a matter of a new DatasetMapper class that has a different __call__ from the default.

2. Models that make use of this new input. We should start with one model at a time. For example, to start with RetinaNet, this involves writing a new RetinaNetWithIgnored class that overwrites the forward() method to process the extra input, e.g. adding some new code between these two lines: https://github.com/facebookresearch/detectron2/blob/5c9e0d9595316ce3447612a4ce1f602911bde2b7/detectron2/modeling/meta_arch/retinanet.py#L163-L164 If that's not enough, maybe it needs to overwrite more methods. But we'll see more when we get there.

[Trouble404]

It's clear, I will try to implement them after the holidays