[FR] Add support for extracting image foregrounds

Ghaleb-alnakhlani commented 2 years ago

Proposal Summary

Download foreground images (images with transparent background) from COCO dataset or Open Images ?

Motivation

What is the use case for this feature? For preparing a new dataset by inserting the extracted objects into a new scene
Why is this use case valuable to support for FiftyOne users in general? Useful in case of preparing an object detection dataset
Why is this use case valuable to support for your project(s) or organization? It will make preparing a computer vision dataset easy.
Why is it currently difficult to achieve this use case? It is not difficult, however, i have not found a feature that does that, specifically, I don`t know how to achieve that using the existing features?

What areas of FiftyOne does this feature affect?

[ x] App: FiftyOne application
[ ] Core: Core fiftyone Python library
[ ] Server: FiftyOne server

Details

There must be a way to get the foreground images, I can see that Fiftyone supports masks and this should lead to the solution. This is very useful for all task that requires dataset preparation.

Willingness to contribute

The FiftyOne Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

[ ] Yes. I can contribute this feature independently.
[ ] Yes. I would be willing to contribute this feature with guidance from the FiftyOne community.
[ x] No. I cannot contribute this feature at this time.

ehofesmann commented 2 years ago

@Ghaleb-alnakhlani COCO and Open Images don't directly provide a way to just download the foreground images. However, you are right that it is fairly straightforward to write a Python loop to extract the masks within FiftyOne and write them to disk.

For example, this script would allow you to extract and write to disk all of the object instances in your dataset separated into folders by their class label:

import os

import numpy as np
import eta.core.image as etai

import fiftyone as fo
import fiftyone.zoo as foz

def extract_classwise_instances(samples, output_dir, label_field, ext=".png"):
    print("Writing extracted objects...")
    for sample in samples.iter_samples(progress=True):
        img = etai.read(sample.filepath)
        img_h,img_w,c = img.shape
        for det in sample[label_field].detections:
            mask = det.mask
            [x,y,w,h] = det.bounding_box
            x = int(x * img_w)
            y = int(y * img_h)
            h, w = mask.shape
            mask_img = img[y:y+h, x:x+w, :] 
            alpha = mask.astype(np.uint8)*255
            alpha = np.expand_dims(alpha, 2)
            mask_img = np.concatenate((mask_img, alpha), axis=2)

            label = det.label
            label_dir = os.path.join(output_dir, label)
            output_filepath = os.path.join(label_dir, det.id+ext)
            etai.write(mask_img, output_filepath)

dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="validation",
    max_samples=10,
    label_types=["segmentations"],
)

output_dir = "/tmp/alpha_segmentations"
label_field = "ground_truth"

extract_classwise_instances(dataset, output_dir, label_field)

I could see a feature like this built into FiftyOne being fairly useful :+1:

Ghaleb-alnakhlani commented 2 years ago

@ehofesmann thank you for this solution. Yes, as you have mentioned it would be very useful to see this feature.

voxel51 / fiftyone