Convert OID to COCO Panoptic GT version

[ozendelait]

Similar to existing boxable version; needed for consolidation/remapping withininst. segm. task

[ozendelait]

Is there somewhere already a version or script to convert OID into COCO panoptic format?

[rodrigob]

Not to my knowledge. A quick google search points to https://github.com/bethgelab/openimages2coco

but that only handles the boxes, not the instance segmentation data.

[michaelisc]

I added code for that conversion to my toolkit: https://github.com/bethgelab/openimages2coco

I think most of it should be fine but I did not test every use case because conversions take a while (pngs have to be loaded and saved from and to disk, that takes time).

There was one problem I ran into: The COCO panoptic format only allows for a single instance to be at one location. Therefore overlapping instances have to be dealt with. For the moment I followed the simple heuristic of placing smaller objects on top of larger objects. This removes almost all errors except for a few strange ones (3 on the val set) that I will look into the following days. There is however the issue that converting COCO panoptic back to OID style annotations will create instance masks with holes where other objects were. I am not sure if we can avoid this with the current data format.

[ozendelait]

Thanks! I'm currently executing the script; for me at validation I got only 3 conflicts: Overlapping masks in image cdbdcb64d5e25fb9 Not in segments: [19849] Not in pixel values: [19849] Overlapping masks in image 5bedf6beab1a83e1 Not in segments: [231, 353, 2332, 6870, 8290, 9556, 13985, 14849, 17680] Not in pixel values: [231, 353, 2332, 6870, 8290, 9556, 13985, 14849, 17680] Overlapping masks in image 30279ed616d417ec Not in segments: [] Not in pixel values: []

I think this small amount of conflicts is acceptable as is?

One question @ script: why are there less annotations than images (validation 42k image; 13k annotations)?; I guess the json includs all images (i.e. also those which have only boxable gt)?

[michaelisc]

I did not get to address the problem with these annotations exactly because I came to the same conclusion. Such a small number of errors could be ok. I'd still fix that though because small errors during data loading can lead to annoying interruptions in the training process if they break things. That should be avoided.

Concerning the number of annotations: Only a subset of the images has segmentation annotations. The panoptic annotation file currently includes information about all images len(oi['images']) = 42k but annotations only for the annotated subset len(oi['annotations']) = 13k. It may be worth changing that to include only the annotated images. And it may also be a good idea to exclude the images that create errors/conflicts as they are few.

[michaelisc]

I just updated the toolkit to fix two things:

• Images without annotations are removed from oi['images'].
• The (few) images with errors during conversion are removed from the dataset. They are still reported though so people are aware of the issue.

[rodrigob]

mmm... I had guessed that the Panoptic format was compatible with the Stuff segmentation format previously used http://cocodataset.org/#format-data which did allow overlap of categories.

We might be able to do "as is" for this version of the competition. But it is semantically wrong, since, for example "button" is considered part of "shirt", and both are usually part of "person". Thus segments in general overlap and are not exclusive (specially when considering large number of categories).

As @michaelisc indicated "small object in front" is a common heuristic. Did you compute how often such overlap occur ?

This removes almost all errors except for a few strange ones (3 on the val set)

What are these "strange" errors ?

COCO panoptic back to OID style annotations will create instance masks with holes where other objects were

This might be a problem for when the participants want to evaluate OID results, if their submissions have "holes" they will lose AP points due to this.

I am not sure if we can avoid this with the current data format.

Is it possible to use an hibrid file format, where both panoptic and object detection elements are present in the json files ? Maybe that could be a way to side-step the issue ?

[ozendelait]

Agreed. Did you keep a list of all conflicting frames? I think the numbers are the same for train right? btw.: the script for Train is currently running but will take ~80h to complete for me :)

[michaelisc]

Did you compute how often such overlap occur ?

3 times on the val set. I did not run conversion of the train set yet. Let's wait for Olivers numbers

What are these "strange" errors ?

I am not sure yet. Will investigate!

Is it possible to use an hibrid file format, where both panoptic and object detection elements are present in the json files ? Maybe that could be a way to side-step the issue ?

That is already the case. Each segments info also contains also the bounding box coordinates.

[michaelisc]

I had a look at the problematic masks. They are simply missing. The mask files are empty (the PNG files are 0 everywhere). I can change the code so empty masks are simply ignored but we can keep the images. Or I can leave it as it is. Any preference @ozendelait ?

[rodrigob]

Each segments info also contains also the bounding box coordinates.

I meant to also include segmentation field (as RLE mask), so that we can encode overlapping per-instance masks, on top of whatever is encoded in the PNG. Ideally things would have per-instance (possibly overlapping) masks, and stuff would only appear in the PNG.

[rodrigob]

They are simply missing.

That would be curious (I would have expected that we filtered out those cases), would be interested on having a count of how many of such cases as encountered.

[michaelisc]

I meant to also include segmentation field (as RLE mask), so that we can encode overlapping per-instance masks, on top of whatever is encoded in the PNG.

Then we could also switch to the original COCO format. If the annotations get too big in that case we can split the dataset up into multiple files. However I think we now already decided to go with the panoptic format for the other datasets. It may be too late to change that. OID will however probably not be the only dataset with this problem.

[ozendelait]

I'd also stick to the panoptic format for now until the dev kits work for all benchmarks (training and submission). After that we can add additional features. @count: I got 4 at validation and 86 in train @empty masks: The clean solution would be to not append the empty/conflicting annotation to the annotations. You can keep the images in the list, of images just as there are the boxable-only images still in the total list.

I updated the downloader, converter and mapping merger which works now for me and OID instance task! I created this patch https://github.com/ozendelait/rvc_devkit/blob/master/segmentation/openimages2coco.patch to support subdirectory structures of train_0 ... train_f for masks as well (both for input and results). @michaelisc: If you like this feature, please add it to your repo

[michaelisc]

@ozendelait I changed the library to exclude only the instances with empty masks instead of the whole image.

I did not include your patch though. After looking at the code I decided that in the long run I'd rather provide a small script that automatically rearranges the files than handling this during conversion. For example I found a hard coded 'train_' in your code that should lead to an error when converting the validation split. It still makes sense to path this for RVC as I am not sure when I will get to write that file rearrangement function. Thanks for figuring out the details of on that and providing the patch!

[ozendelait]

Thanks! I think we can close this issue for good right? @ "train_": yes, as val is only a single directory and works out of the box (that hacky line only triggers if the folder is not found). train is the only split missing. I like the split into 16 subdirs instead of a single folder; I'll update the patch to work with your newest version.