[FR] Add support for keypoint skeleton annotation in CVAT

[ehofesmann]

CVAT recently added a way to annotate pre-defined keypoint skeletons. This new label type should be incorporated into our CVAT integration using existing KeypointSkeletons on a FiftyOne dataset.

[stevezkw1998]

There is a use case: We have two field: prediction, prediction-pose prediction: bbox prediction-pose: skeleton(with edges)

Then we want to upload both of them to CVAT

[smidm]

dataset.annotate() sends fo.Keypoints annotations and/or predictions to CVAT as set of point annotations without associated skeleton and keypoint names (as of FiftyOne version 0.21). This in many usecases unusable.

A workaround is to export the dataset to COCO Keypoints format and import it in CVAT. Unfortunately CVAT is rather picky on import.

Here is how to do it.

export dataset with keypoint predictions in keypoint_prediction from FiftyOne:

dataset.export(
    export_dir="./sample_dataset/",
    dataset_type=fo.types.COCODetectionDataset,
    label_field="keypoint_prediction",
)

in CVAT:

1. create new task and upload images
   1. create new task
   2. upload images
2. setup skeleton
   1. open project or tasks
   2. click setup skeleton
   3. create skeleton with the same keypoint names as in fo.KeypointSkeleton in fiftyone dataset
   4. name the skeleton as the fo.Keypoint label (important!)
3. fix fiftyone export for CVAT using code bellow
4. upload fixed annotations
   1. task -> action -> upload annotations
   2. import format: COCO Keypoints
   3. select fixed labels file

If anything goes wrong: make a dummy dataset with skeleton label, few images and annotations in CVAT, export it to COCO Keypoints format and compare with the file you want to import.

import json

labels_in = "out/ankle_500/labels.json"
labels_fixed_for_cvat = "out/ankle_500/labels_fixed_for_cvat.json"
KEYPOINT_NAMES = ['nose', 'left_eye',...]

def iterable_to_int(t):
    return [int(x) for x in t]

with open(labels_in, "r") as f:
    labels = json.load(f)

# CVAT expects categories to start at 1, requires keypoints and skeleton to be defined
for category in labels["categories"]:
    category["id"] += 1
    category["keypoints"] = KEYPOINT_NAMES
    category["skeleton"] = []

# CVAT expects annotations to have a bbox, area, attributes, segmentation and no score
for annotation in labels["annotations"]:
    annotation["category_id"] += 1    
    keypoints = np.array(annotation["keypoints"]).reshape(-1, 3)[:, :2]
    tl = keypoints.min(axis=0)
    br = keypoints.max(axis=0)
    annotation["bbox"] = iterable_to_int(tuple(tl) + tuple(br - tl))
    annotation["area"] = int((br[0] - tl[0]) * (br[1] - tl[1]))
    annotation["attributes"] = {
        "occluded": False
    }
    annotation["segmentation"] = []
    del annotation["score"]

with open(labels_fixed_for_cvat, "w") as f:
    json.dump(labels, f)

[bc-arl]

Any updates on this feature? Would like to use it.

[stevezkw1998]

I have a humble workaround idea (similar but different with @smidm 's helpful idea) regarding to this issue 'upload skeleton to CVAT':

1. Prepared skeleton labels for samples in your dataset.
2. Upload pure images to CVAT and create a project.
3. Draw the skeleton maually on the setup skeleton on CVAT project level.
4. Export dataset skeleton labels xml in CVAT format and do some neccesary format transformation, like xml keypoints -> skeleton
5. Use _cvatsdk to task.import_annotations(xml_path).
6. Then you can have your skeleton labels with edge on CVAT task.

To download it back:

1. Use cvat api to download the exported annoations results xml back to some place in local.
2. The downloaded xml only has keypoints, but it doesn't matter.
3. Do some neccesary format transformation, and you can have your skeleton schema/definition on your original dataset.
4. Add those keypoint(skeleton) to the samples respectively.
5. Then theoretically you can visualize the skeleton as annoations results from CVAT on FiftyOne app.

If that works, hope someday it will be integrated into FiftyOne, thanks!

[RolandWolman]

I followed @smidm instructions and successfully imported the keypoint labels into CVAT. Do you also have a suggestion on how to best import the fixed labels back to fiftyone in this specific workflow?

[smidm]

Export dataset from CVAT: Tasks -> Actions -> Export task dataset, set "Export format" to "COCO Keypoints". It may take few minutes to prepare the export, be patient.

load in fiftyone:

dataset = fo.Dataset.from_dir(
    data_path="<images dir>",
    labels_path="<CVAT exported json file>",
    dataset_type=fo.types.COCODetectionDataset,
    label_field="annotation",
)
# set COCO or custom skeleton
dataset.skeletons["annotation"] = fo.KeypointSkeleton(
        labels=
[
    'nose',
    'left_eye',
    'right_eye',
    'left_ear',
    'right_ear',
    'left_shoulder',
    'right_shoulder',
    'left_elbow',
    'right_elbow',
    'left_wrist',
    'right_wrist',
    'left_hip',
    'right_hip',
    'left_knee',
    'right_knee',
    'left_ankle',
    'right_ankle'
],
        edges=
[
    [15, 13], [13, 11], [16, 14], [14, 12],  # legs
    [11, 12], [5, 11], [6, 12],  # upper body
    [5, 6], [5, 7], [6, 8], [7, 9], [8, 10],  # arms, shoulders
    [1, 2], [0, 1], [0, 2], [1, 3], [2, 4], [3, 5], [4, 6],  # head
],
)

Beware that the keypoint visibility data is lost on import, see https://github.com/voxel51/fiftyone/issues/1581

You can try https://github.com/openvinotoolkit/datumaro for non-destructive dataset processing.

First store the data in COCO directory structure as described in https://openvinotoolkit.github.io/datumaro/latest/docs/data-formats/formats/coco.html#import-coco-dataset

└─ Dataset/
    ├── images/
    │   ├── <image_name1.ext>
    │   ├── <image_name2.ext>
    │   └── ...
    └── annotations/
        ├── coco_person_keypoints.json
        └── ...

and load into datumaro:

dataset = dm.Dataset.import_from(
    "<dataset dir>",
    'coco_person_keypoints',
)

[medphisiker]

Export dataset from CVAT: Tasks -> Actions -> Export task dataset, set "Export format" to "COCO Keypoints". It may take few minutes to prepare the export, be patient.

Hello, thank you for the detailed instructions and the code.

I did the steps you indicated and received a dataset in FiftyOne with two fields:

Name:        fish-points-dataset
Media type:  image
Num samples: 1
Persistent:  True
Tags:        []
Sample fields:
    id:                    fiftyone.core.fields.ObjectIdField
    filepath:              fiftyone.core.fields.StringField
    tags:                  fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:              fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    annotation_detections: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    annotation_keypoints:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Keypoints)

Everything is correct =) I also tried exporting this dataset to CVAT for labeling.

# upload FiftyOne's dataset to CVAT for labeling
anno_key = "dets_run"

anno_results = dataset.annotate(
    anno_key,
    backend="cvat",
    url="my_local_cvat_ip",
    username="admin-username",
    password="admin-password",
    label_field="annotation_detections", # there we need to set only one field
    segment_size=100
    allow_additions=True,
    allow_deletions=True,
    allow_label_edits=True,
    allow_spatial_edits=True,
    launch_editor=True,
)

But when I do it I should to chose one field from two:

• annotation_detections with bboxes
• annotation_keypoints with keypoints

Each method works correctly, it loads either bbox or key points. But I need both.

Is there a way to upload annotations from both fields of the two Fifty One's dataset to CVAT?

[medphisiker]

There is also another interesting feature. Initially, in CVAT, we set the skeleton entity:

After importing from FiftyOne to CVAT, you get just a set of three key points by this code:

# upload FiftyOne's dataset to CVAT for labeling
anno_key = "dets_run"

anno_results = dataset.annotate(
    anno_key,
    backend="cvat",
    url="my_local_cvat_ip",
    username="admin-username",
    password="admin-password",
    label_field="annotation_keypoints",
    label_type="keypoints",
    segment_size=100,
    allow_additions=True,
    allow_deletions=True,
    allow_label_edits=True,
    allow_spatial_edits=True,
    launch_editor=True,
)

Is there a way to import key points into CVAT like CVAT's skeleton? )