search

voxel51 / fiftyone

Refine high-quality datasets and visual AI models
https://fiftyone.ai
Apache License 2.0
8.84k stars 557 forks source link

[FR] Add support for exporting COCO boxes and keypoints together #2958

Open jaderxnet opened 1 year ago

jaderxnet commented 1 year ago

System information

Commands to reproduce

As thoroughly as possible, please provide the Python and/or shell commands used to encounter the issue. Application steps can be described in the next section.

export_dir = "/content/gdrive/MyDrive/Colab Notebooks/Create dataset with FiftyOne/coco-detection-dataset"
list_label = ["keypoints","bbox"]  # for example
print(list_label)
print(type(list_label))
# Export the dataset
dataset.export(
    export_dir=export_dir,
    dataset_type=fo.types.COCODetectionDataset,
    label_field=list_label,
)

Describe the problem

I need to convert a Brace (Link) based dataset to COCO Dataset format(Detection and Keyipoints). I managed to do a test with FiftyOne on Google Colab loading Bounding Box(Detection) and Keypoints (Link). But when I try to export to COCO format, I can only export Detection or Keypoints, not both together like two annotations on same image. In the documentation it is possible that the label_fild parameter is a list(link), but when I sent label_fild as a list, I got this error. I saw the implementation and maybe the lebel_field as a list is not implemented. Did I do something wrong? Can anyone help me with this?

Code to reproduce issue

Google Colab Link

ERROR LOGs

['keypoints', 'bbox']
<class 'list'>
Directory '/content/gdrive/MyDrive/Colab Notebooks/Create dataset with FiftyOne/coco-detection-dataset' already exists; export will be merged with existing files
WARNING:fiftyone.core.collections:Directory '/content/gdrive/MyDrive/Colab Notebooks/Create dataset with FiftyOne/coco-detection-dataset' already exists; export will be merged with existing files
   0% |/----------------|   0/153 [45.2ms elapsed, ? remaining, ? samples/s] 
INFO:eta.core.utils:   0% |/----------------|   0/153 [45.2ms elapsed, ? remaining, ? samples/s] 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-23-d64fdd810875>](https://localhost:8080/#) in <cell line: 6>()
      4 print(type(list_label))
      5 # Export the dataset
----> 6 dataset.export(
      7     export_dir=export_dir,
      8     dataset_type=fo.types.COCODetectionDataset,

5 frames
[/usr/local/lib/python3.10/dist-packages/fiftyone/core/collections.py](https://localhost:8080/#) in export(self, export_dir, dataset_type, data_path, labels_path, export_media, rel_dir, dataset_exporter, label_field, frame_labels_field, overwrite, **kwargs)
   7948 
   7949         # Perform the export
-> 7950         _export(
   7951             self,
   7952             export_dir=export_dir,

/usr/local/lib/python3.10/dist-packages/fiftyone/core/collections.py in _export(sample_collection, export_dir, dataset_type, data_path, labels_path, export_media, rel_dir, dataset_exporter, label_field, frame_labels_field, overwrite, **kwargs)
  10418 
  10419     # Perform the export
> 10420     foud.export_samples(
  10421         sample_collection,
  10422         dataset_exporter=dataset_exporter,

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/data/exporters.py](https://localhost:8080/#) in export_samples(samples, export_dir, dataset_type, data_path, labels_path, export_media, rel_dir, dataset_exporter, label_field, frame_labels_field, num_samples, **kwargs)
    354         )
    355 
--> 356     write_dataset(
    357         samples,
    358         sample_parser,

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/data/exporters.py](https://localhost:8080/#) in write_dataset(samples, sample_parser, dataset_exporter, num_samples, sample_collection)
    414         (UnlabeledImageDatasetExporter, LabeledImageDatasetExporter),
    415     ):
--> 416         _write_image_dataset(
    417             dataset_exporter,
    418             samples,

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/data/exporters.py](https://localhost:8080/#) in _write_image_dataset(dataset_exporter, samples, sample_parser, num_samples, sample_collection)
    894 
    895                     # Export sample
--> 896                     dataset_exporter.export_sample(
    897                         image_or_path, label, metadata=metadata
    898                     )

[/usr/local/lib/python3.10/dist-packages/fiftyone/utils/coco.py](https://localhost:8080/#) in export_sample(self, image_or_path, label, metadata)
    824             labels = label.keypoints
    825         else:
--> 826             raise ValueError(
    827                 "Unsupported label type %s. The supported types are %s"
    828                 % (type(label), self.label_cls)

ValueError: Unsupported label type <class 'dict'>. The supported types are (<class 'fiftyone.core.labels.Detections'>, <class 'fiftyone.core.labels.Polylines'>, <class 'fiftyone.core.labels.Keypoints'>)

What areas of FiftyOne does this bug affect?

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

brimoor commented 1 year ago

Unfortunately FiftyOne's builtin COCO exporter does not currently support merging data from multiple label fields (eg one Detections and one Keypoints) into a single JSON file on-the-fly.

jaderxnet commented 1 year ago

Thanks a lot for the quick response. I suggest updating the documentation description to make this clear. As soon as possible, I will collaborate for this feature to be implemented because I am an admirer of your work.

brimoor commented 1 year ago

Thanks for the kind words!

I think the way we can support this is by adding an optional keypoints parameter to COCODetectionDatasetExporter that is similar to iscrowd that allows the user to specify that their Detection instances have a an attribute that contain Keypoint instances that they'd like to include in the export. This attribute can be parsed here: https://github.com/voxel51/fiftyone/blob/de89c6d45669bf31489b17a4dd3b6bb59b4e5757/fiftyone/utils/coco.py#L1263

Assuming Dataset is properly constructed, the final syntax would then be like this:

dataset.export(
    labels_path="/path/for/coco.json",
    dataset_type=fo.types.COCODetectionDataset,
    label_field="<detetections_field>",
    keypoints="<keypoints_attribute>",
)

The problem with storing Detection and Keypoint in separate fields and passing them via the label_field argument is that there is no official way to know that two objects correspond to each other. I imagine users would prefer to store the keypoints as an attribute of the bounding box to keep the data better organized (we should add support for rendering such data in the FiftyOne App, too!)

jaderxnet commented 1 year ago

In fact, I'm specifically talking about files like person_keypoints_val2017.json, which have a structure something like this:

Array de annotations with: --- id --- segmentations Array with N Segmentations --- num_keypoints related with Segmentation --- area related with Segmentation --- keypoints Array with 17 Keypoints and confidence in flat 17 x 3 = 51 like [x,y,c,x,y,c,...] --- bbox Detection --- imageId --- category_id --- iscrowd ...

and each image can have multiple annotations. So, it seems to be a very different pattern from the tool's pattern. See a part of the file person_keypoints_val2017.json.

"annotations": [{"segmentation": [[125.12,539.69,140.94,522.43,100.67,496.54,84.85,469.21,73.35,450.52,104.99,342.65,168.27,290.88,179.78,288,189.84,286.56,191.28,260.67,202.79,240.54,221.48,237.66,248.81,243.42,257.44,256.36,253.12,262.11,253.12,275.06,299.15,233.35,329.35,207.46,355.24,206.02,363.87,206.02,365.3,210.34,373.93,221.84,363.87,226.16,363.87,237.66,350.92,237.66,332.22,234.79,314.97,249.17,271.82,313.89,253.12,326.83,227.24,352.72,214.29,357.03,212.85,372.85,208.54,395.87,228.67,414.56,245.93,421.75,266.07,424.63,276.13,437.57,266.07,450.52,284.76,464.9,286.2,479.28,291.96,489.35,310.65,512.36,284.76,549.75,244.49,522.43,215.73,546.88,199.91,558.38,204.22,565.57,189.84,568.45,184.09,575.64,172.58,578.52,145.26,567.01,117.93,551.19,133.75,532.49]],"num_keypoints": 10,"area": 47803.27955,"iscrowd": 0,"keypoints": [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,142,309,1,177,320,2,191,398,2,237,317,2,233,426,2,306,233,2,92,452,2,123,468,2,0,0,0,251,469,2,0,0,0,162,551,2],"image_id": 425226,"bbox": [73.35,206.02,300.58,372.5],"category_id": 1,"id": 183126},
rusmux commented 1 year ago

Are there any updates on this? The documentation is highly misleading and forces to dig into source code. At least let update the documentation

julled commented 1 year ago

It would be also nice to export boxes and segmentations at the same time.

smidm commented 11 months ago

I'm using following to add bboxes to the exported COCO dataset with keypoints only. It's just a bbox inferred from the keypoints, but it works for me.

import numpy as np
import json

def fix_coco(
        input_json,
        output_json,
):
    def iterable_to_int(t):
        return [int(x) for x in t]

    with open(input_json, "r") as f:
        coco_keypoints = json.load(f)

    for annotation in coco_keypoints["annotations"]:
        keypoints = np.array(annotation["keypoints"]).reshape(-1, 3)[:, :2]
        tl = keypoints.min(axis=0)
        br = keypoints.max(axis=0)
        if 'bbox' not in annotation:
            annotation["bbox"] = iterable_to_int(tuple(tl) + tuple(br - tl))
        if 'area' not in annotation:
            annotation["area"] = int((br[0] - tl[0]) * (br[1] - tl[1]))

    with open(output_json, "w") as f:
        json.dump(coco_keypoints, f)