OIDv5 - Segmentation Masks Support

Auth0rM0rgan commented 4 years ago

Hey @monocongo, Thanks for sharing your code!

I would like to use OIDv5 segmentation masks to train my instance segmentation approach with OIDv5. It would be nice to have:

[ ] Toolkits to download OIDv5 segmentation masks.
[ ] Convert OIDv5 segmentation annotation files to COCO Json format.

Thanks!

monocongo commented 4 years ago

Thanks @Auth0rM0rgan

If you can please post a link to documentation that describes the COCO JSON format for segmentation, so I can use that as a guide. I have used COCO in the past and may have some old code to use as a guide, but it was for bounding boxes and not segmentation mask polygons.

Related to cvdata issue #145

monocongo commented 4 years ago

Create COCO Annotations from Scratch

woutgg commented 4 years ago

Hi, thanks for sharing this project!

As I needed segmentation masks with the images, I forked your repo and continued the work you had already done. Apparently there is no file-level hosting of the mask files, so the script downloads all the segmentation zip files (about 4.4GB) and extracts from them what is actually needed.

I've also added some more progress bars and log messages since (with my internet connection) the initial wait time was quite long and I wanted to have a bit more progress info available.

If you are interested, please have a look at my fork. I'd be happy to open a PR and rework things where necessary.

woutgg commented 4 years ago

BTW, I realize this issue has been closed, but I was under the impression that this might be a good place to continue on the matter anyway. Let me know if you disagree.

monocongo commented 4 years ago

@woutgg this is very welcome news, thanks for your work on this!

BTW I have noticed that there's now an OID version 6. Do you know what the differences are in terms of how this new version may affect this code? For example, have any download URLs changed, etc.?

If you can further summarize what has changed and/or been added in your fork then please update here in the comments (I'm lazy!).

Thanks again for your efforts, quite helpful.

woutgg commented 4 years ago

BTW I have noticed that there's now an OID version 6. Do you know what the differences are in terms of how this new version may affect this code? For example, have any download URLs changed, etc.?

I was also wondering this and I do not think it affects the code at all. In their introductory text here, this is the main difference: "In V6 we added 507k localized narratives: multimodal descriptions of images consisting of synchronized voice, text, and mouse traces over the objects being described."

The class names, most bbox CSVs as well as the segmentation files still point to V5; the exception to this is the train bbox CSV but I didn't check the difference yet.

If you can further summarize what has changed and/or been added in your fork then please update here in the comments (I'm lazy!).

Sure :), here goes:

Commit 1 fixes some small errors in the preliminary changes you made.
Commit 2 refactors the HTTP downloads into a function that runs a progress bar along (mainly for the large training bbox+segmentation CSVs) and adds a few 'heartbeat' log messages.
Commit 3 adds the mask downloading. This happens by first fetching the zip files for each split section (16, as they are binned to the first hex char), then each masking image as listed in the CSV is extracted from the appropriate archive and moved into place. The zip files are opened only once per class label iteration, to get decent performance - this could be improved to opening them only once per split section.

I still have to double check if the resultant file output is acutally correct, so beware of bugs.

woutgg commented 4 years ago

So apparently the output was not correct, I had overlooked that your code filters on the bbox attributes (IsOccluded etc.). I've added a commit that fixes this, and another that skips downloading of existing files.

This time I checked the file counts created by the script against manual counts (well with some help from awk and friends) and they match up.

monocongo commented 4 years ago

Good catch. We should probably add options for keeping or filtering bboxes with the IsOccluded, IsGroupOf, etc. flags.

woutgg commented 4 years ago

Indeed, my thought as well. It's even on my todo-list already, but since I don't need it at the moment I'm leaving it for now.

If you want me to turn the current changes into a PR, just let me know.

monocongo commented 4 years ago

Yes, absolutely, you've made some good contributions here that I'd like to be folded into the main repository. Thanks!

woutgg commented 4 years ago

There you go!

I'm currently trying out a model implementation that uses OID annotations, but in case that fails I might implement the COCO conversion as well.

Rm1n90 commented 4 years ago

Hey @woutgg, @monocongo,

I would like to ask you that do you have any plan to support to download OID segmentation mask and segmentation annotations in Coco format?

monocongo commented 4 years ago

I currently don't have the bandwidth to devote to this conversion to the COCO format. Surprisingly there doesn't seem to be such a converter available as an open source tool, but I have not done an exhaustive search. I have started work on this over in my computer vision datasets manipulation package repository, cvdata (see this issue), and that seems to be a more logical place for a conversion such as this, but no immediate plans to complete that soon. PRs are very welcome!

nicolasmetallo commented 4 years ago

Thanks for the excellent work on this library! I did a converter some time ago from OID to PASCAL VOC here: https://github.com/nicolasmetallo/train_custom_detection_with_OID maybe some of it it's useful. There are many available converters from PASCAL to COCO.

monocongo / openimages

OIDv5 - Segmentation Masks Support #7