How to annotate person keypoints in COCO style

[mgarbade]

I try to annotate my own images in the same way that as the COCO person-keypoint-detection challenge. Meaning I just want to annotate nose, neck, shoulder etc of persons. So far I

• created a Dataset
• created a Category "person"
• added my own images to coco-annotator/datasets/myDataset
• imported the images by pressing "Scan"

Now when I select an image, I can see the Category "person" on the right. So I can for example draw a polygon around the person. However when I select "Keypoints Tool" I can place keypoints, but not specify "what keypoint" (nose or neck or shoulder) The respective Dropdown menu in "Settings for next Keypoint - Label" section is empty. How can I fill it?

Update When opening an image there is a "settings" symbol next to the "person" class if you select it, you can manually enter the coco-keypoint labels

nose                        
eye_left                  
eye_right                
ear_left                   
ear_right                 
shoulder_left          
shoulder_right         
elbow_left               
elbow_right             
wrist_left                 
wrist_right               
hip_left                    
hip_right                
knee_left               
knee_right             
ankle_left              
ankle_right            

When exporting the labels, the "skeleton" values of the exported json file do not correspond to the "skeleton" used in the original coco-annotations. So I just manually replaced it.

[mgarbade]

Getting closer: This is how to add the keypoint categories to the "Person" category.

I still need to click keypoints, even if they are not visible, not sure what to do...

• mark them as "not visible" and try to estimate there location
• mark them as "not labeled" - I still need to click somewhere using this option, which is weird

question is if I need to click every keypoint even if it's not visible at all / or being exactly occluded by another keypoint.

At the current state it seems that I need to annotate all 17 keypoints with there respective flag (LABELED_VISIBLE, NOT_LABELED, LABELED_NOT_VISIBLE) in one shot, which has the following problems:

• keypoints that are occluded and fall on the exact same location as another (eg a person standing sideways, while the left eye is visible and the right eye is occluded but probably at the exact same location)
  • also the need to switch between visible / not-visible for every keypoint for a side-standing person, without keyboard shortcut is tedious.

I keep on digging...

[jsbroks]

You are for right, the current design for keypoints is not very intuitive and tedious. Do you have any suggestions to implemented key points in a cleaner way?

You are right, two keypoints 'could' fall on the same pixel. You can solve this problem by placing the keypoints very close to each other at max zoom in. This way you are able to click on the correct key point

[mgarbade]

Well, I guess that keyboard shortcuts cut help here. Eg pressing 0 for not visible which stores then (0,0,0) in the corresponding json, while automatically moving to the annotation of the next keypoint.

At least this is how it seems to be done in the original coco - person keypoint annotations. Here is an example from coco/person_keypoint_val2017.json:

"num_keypoints": 13
"area": 68822.32895
"iscrowd": 0
"keypoints": [389
97
2
[...]
413
408
1
316
399
2
0
0
0
[...]
0
0
0]

which seems to consist of triplets (pixel-coordinate-x , pixel-coordinate-y, visibility-flag)

[mgarbade]

@jsbroks Could you give me a quick hint, where / how to add such a functionality? (I would drastically speed up my annotation time)

I assume I need an entry in shortcuts.js which listens to the 0 key being pressed on the keyboard and then calling some method to add a (0,0,0) keypoint (i.e. a keypoint with 0 coordinates and 0 -visibility-flag).

    {
      default: ["0"],
      name: "Skip non visible keypoint",
      function: () => {
        if (this.currentAnnotation) {
          let currentKeypoint = this.currentAnnotation.currentKeypoint;
          if (currentKeypoint) {
            this.currentAnnotation.keypoints.skipKeypoint(currentKeypoint)
          } 
        }
      }
    },

and in keypoints.js I added the following function:

  skipKeypoint(keypoint) {
    keypoint.keypoints = this;
    keypoint.path.keypoints = this;
    keypoint._visibility = keypoint.LABELED_NOT_VISIBLE;
    keypoint.color = this.strokeColor;
    keypoint.path.strokeWidth = this.strokeWidth;

    let indexLabel = keypoint.indexLabel;
    if (this._labelled.hasOwnProperty(indexLabel)) {
      keypoint.indexLabel = -1;
    } else {
      this._labelled[indexLabel] = keypoint;
    }

    this._keypoints.push(keypoint);
    this.addChild(keypoint.path);
    this._drawLines(keypoint);
    keypoint.path.bringToFront();
  }

However, this is not working so far ... (I'm also no expert on js or vue)

[AhmedGhazale]

@mgarbade this would be very helpful for me ...so if you managed to find a good solution please share it :)

[jsbroks]

@mgarbade If I understand this correctly when the user clicks 0, you want to create a key point at [0,0] with zero visibility?

Even if the key point isn't visible, shouldn't it still have coordinates?

[mgarbade]

Exactly. As mentioned above this seems to be the practice that was applied in the official COCO person keypoint annotation.

Best would be to be able to select all visibility flags via keyboard shortcuts: "0", "1", "2" , where "0" is "NOT_LABELED" -> (0, 0, 0) "1" is "LABELED_BUT_NOT_VISIBLE" ->(x-pos, y-pos, 1) ((seems to be applied when a keypoint is occluded by another person "2" is "LABELED_VISIBLE" -> (x-pos, y-pos, 2)

That seems to be the fastest way for annotating.

As I'm trying to dig trough your code, a hint would be already helpful (in case you don't agree with my suggestion / don't have time to implement it)

[jsbroks]

Sadly I won't have time to implement this. You're on the right track but we can simplify this.

Note: Code is untested

shortcuts.js

    {
      default: ["0"],
      name: "Skip non visible keypoint",
      function: () => {
        if (this.currentAnnotation) {
          this.currentAnnotation.addKeypoint(new paper.Point(0,0), 0);         
        }
      }
    },

[mgarbade]

Your suggested coded combined with import paper from "paper"; in the top of shortcuts.js, works exactly once. Pressing the shortcut another time, has no effect though. Still this is already close to the solution, so thx so far.

[jsbroks]

You'll have to debug to figure out why thats happening. I'm not sure the cause

[mgarbade]

So the problem comes from the fact, that while pressing "0" launches the correct function, it still draws a keypoint onto the image. This however prevents the shortcut to be executed another time as it would draw ontop of an existing point due to this line:

    addKeypoint(point, visibility, label) {
      if (label == null && this.keypoints.contains(point)) return;
      [...]

Any idea how I could add such an "empty Keypoint" without drawing it?

[mgarbade]

So I added a function called "skipKeypoint" to "keypoint.js"

  skipKeypoint(keypoint) {
    console.log("keypoints.js/skipKeypoint")
    this._keypoints.push(keypoint);
  }

and another "skipKeypoint" to "Annotation.vue":

    [...]
    skipKeypoint(point, visibility, label) {
      console.log("Annotation.vue/skipKeypoint")
      let keypoint = new Keypoint(0, 0, {
              visibility: 0,
              indexLabel: 0});
      this.keypoints.skipKeypoint(keypoint);
    },

and in "shortcut.js"

          default: ["0"],
          name: "Skip non visible keypoint",
          function: () => {
            this.currentAnnotation.skipKeypoint(new paper.Point(0, 0), 0);
            console.log("pressed 0 in shortcuts.js")
          }
        },

However two functionalities are missing:

• It still draws some kind of point on the image (which is undesired)
• It does not switch to the next keypoint name in the list of keypoints

Could you tell me, where to find the above mentioned functionalities, so I can fix that?

[mgarbade]

I could not make the skipKeypoint functionality working properly, however a friend who knows some vuejs could implement shortcuts to easily switch between the three visibility flags: see the solution in this commit

[jsbroks]

@mgarbade could you send me an email

[jsbroks]

@mgarbade I suggest you continue with what I stated earlier and debug this function to see why it works the first time but not the second.

• Is it calling the function every time?
• Is the function on the correct annotation?

All the files in the shortcuts.js are mixin with the Annotator.vue file. shorcuts.js can access all the computed properties, methods and data of Annotator.vue

[mgarbade]

To make your suggested shortcut

{
  default: ["+"],
  name: "Skip non visible keypoint",
  function: () => {
    if (this.currentAnnotation) {
      this.currentAnnotation.addKeypoint(new paper.Point(0,0), 0);         
    }
  }
},

work every time, one could edit this line from

addKeypoint(point, visibility, label) {
  if (label == null && this.keypoints.contains(point)) return;
  visibility = visibility || parseInt(this.keypoint.next.visibility);

to

  addKeypoint(point, visibility, label) {
  if (label == null && this.keypoints.contains(point) && visibility != 0) return;
  if (visibility != 0){
    visibility = visibility || parseInt(this.keypoint.next.visibility);
  }

The line obviously checks whether addKeypoint tries to place the new (0,0,0) keypoint at a coordinate which already contains a keypoint. This is the case, when pressing the shortcut twice.

The visibility != 0 hack fixes that, but it's very ugly. Also there appears a black keypoint in the center of the image at (0,0).

I guess this raises a general question: If I see it correctly, keypoints with visibilty flag "0" should not be drawn at all, right? They should be treated in the same way as "deleted keypoints" or keypoints that have never been annotated.

[jsbroks]

Pass in

new paper.Point(0-this.image.raster.width/2, 0-this.image.raster.height/2)

This should place it at top left.

Visibility we can discuss in a different issue.

[mgarbade]

This is working! Thx a lot. I further added some lines to suppress the drawing of "NOT_LABELED" keypoints as can be seen from this commit (based on Matt Deboer's last commit aa78c426fe98ad948aa85048b76508d4f34d10c5 Date: Tue Jan 7 17:26:01 2020 -0800 )