Allow keypoints to have custom metadata / attributes

[asymingt]

Is your feature request related to a problem? Please describe. I'd like to be able to annotate an image with many keypoints, and assign to each keypoint a set of values / attributes, some of which are optional. For example, consider tagging a picture of Swiss cheese with keypoints representing the hole locations. We could label each hole as acceptable or unacceptable, but we might also want to capture aspects of the hole alongside the labeling. For example, numeric values representing the breadth and depth of the hole, or a text label for some notes about it.

Describe the solution you'd like It would be great to be able to have something like this supportable in the label template:

<View>
  <KeyPointLabels name="kp-1" toName="img-1">
    <Label value="acceptable" background="#D4380D"/>
    <Label value="unacceptable" background="#0fb503"/>
    <!-- Attributes -->
    <Number name="depth" />
    <Number name="breadth" />
    <TextArea name="notes"></TextArea>
  </KeyPointLabels>
  <Image name="img-1" value="$img" zoom="true" zoomControl="true" rotateControl="false"/>
</View>

Which then ultimately get mapped to something like this:

{
    "original_width": 2000,
    "original_height": 2000,
    "image_rotation": 0,
    "value": {
        "x": 4.282073,
        "y": 79.588539,
        "width": 0.25,
        "keypointlabels": [
            "unacceptable"
        ]
    },
    "meta": {
        "depth": 0.25,
        "breadth": 0.05,
        "notes": "This would create a perception of low value in the cheese market"
    },
    "id": "LABEL1",
    "from_name": "kp-1",
    "to_name": "img-1",
    "type": "keypointlabels"
}

In the user interface the field themselves will be editable when an keypoint is clicked -- a panel comes up to edit attributes. The immutable values will be shown under each label in the list of labels to the right of the image, when the label is clicked and its detail is unfolded. This will keep the UI sparse, and allow attribute modification on-demand.

Describe alternatives you've considered One could add a different <KeyPointLabels> for each attribute and then have three overlapping keypoints. But this is a lot to manage. The per-image use of <Number> or <TextArea> would also not work without knowing how many keypoints are going to be added. And then you need to manage the correspondences between the keypoints and this per-image data.

Additional context This is likely related to https://github.com/heartexlabs/label-studio/issues/141.

[Gondragos]

Hey, @asymingt

We actually have a perRegion attribute which looks pretty similar to what you are looking for. For example, you can use this configuration:

<View>
  <KeyPointLabels name="kp-1" toName="img-1">
    <Label value="acceptable" background="#D4380D"/>
    <Label value="unacceptable" background="#0fb503"/>
  </KeyPointLabels>
  <Image name="img-1" value="$img" zoom="true" zoomControl="true" rotateControl="false"/>

  <View visibleWhen="region-selected">
    <Header>Depth:</Header>
    <Number name="depth" toName="img-1" perRegion="true"/>
    <Header>Breadth:</Header>
    <Number name="breadth" toName="img-1" perRegion="true"/>
    <Header>Notes:</Header>
    <TextArea name="notes" toName="img-1" perRegion="true"></TextArea>
  </View>
</View>

In this case, the results will look like this:

[
  {
    "original_width": 2000,
    "original_height": 2000,
    "image_rotation": 0,
    "value": {
      "x": 4.282073,
      "y": 79.588539,
      "width": 0.25,
      "keypointlabels": [
        "unacceptable"
      ]
    },
    "id": "LABEL1",
    "from_name": "kp-1",
    "to_name": "img-1",
    "type": "keypointlabels",
    "origin":"manual"
  },
  {
    "original_width": 2000,
    "original_height": 2000,
    "image_rotation": 0,
    "value": {
      "x": 4.282073,
      "y": 79.588539,
      "width": 0.25,
      "number": 0.25
    },
    "id": "LABEL1",
    "from_name": "depth",
    "to_name": "img-1",
    "type": "number",
    "origin": "manual"
  },
  {
    "original_width": 2000,
    "original_height": 2000,
    "image_rotation": 0,
    "value": {
      "x": 4.282073,
      "y": 79.588539,
      "width": 0.25,
      "number": 0.05
    },
    "id": "LABEL1",
    "from_name": "breadth",
    "to_name": "img-1",
    "type": "number",
    "origin": "manual"
  },
  {
    "original_width": 2000,
    "original_height": 2000,
    "image_rotation": 0,
    "value": {
      "x": 4.282073,
      "y": 79.588539,
      "width": 0.25,
      "text": [
        "This would create a perception of low value in the cheese market"
      ]
    },
    "id": "LABEL1",
    "from_name": "notes",
    "to_name": "img-1",
    "type": "textarea",
    "origin": "manual"
  }
]

There is one result for each control tag (<Label/>, <Number/>, <TextArea/>), but they all have the same ID as long as they are associated with the same region.

Does it solve your issue?

[asymingt]

Wow, this is pretty close to my suggested approach, albeit a bit more verbose in JSON. The only trouble is that I think <Number> is broken in this pattern. For example, this works nicely in the label-studio playground:

<View>
  <Image name="img" value="$image" zoom="true" zoomControl="true"/>
  <KeyPointLabels name="labels" toName="img" strokewidth="2" opacity="1" >
      <Label value="Engine" background="red"/>
      <Label value="Tail" background="blue"/>
  </KeyPointLabels>
  <View visibleWhen="region-selected">
    <Header>Rating:</Header>
    <Rating name="rating" toName="img" maxRating="10" icon="star" size="medium" perRegion="true"/>
    <Header>Gender:</Header>
    <Choices name="gender" toName="img" choice="single-radio" perRegion="true">
      <Choice alias="M" value="Male" />
      <Choice alias="F" value="Female" />
      <Choice alias="NB" value="Nonbinary" />
      <Choice alias="X" value="Other" />
    </Choices>
  </View>
</View>

But adding <Number> to the region-selected view breaks the playground engine:

<View>
  <Image name="img" value="$image" zoom="true" zoomControl="true"/>
  <KeyPointLabels name="labels" toName="img" strokewidth="2" opacity="1" >
      <Label value="Engine" background="red"/>
      <Label value="Tail" background="blue"/>
  </KeyPointLabels>
  <View visibleWhen="region-selected">
    <Header>Number:</Header>
    <Number name="number" toName="img" perRegion="true"/>
    <Header>Rating:</Header>
    <Rating name="rating" toName="img" maxRating="10" icon="star" size="medium" perRegion="true"/>
    <Header>Gender:</Header>
    <Choices name="gender" toName="img" choice="single-radio" perRegion="true">
      <Choice alias="M" value="Male" />
      <Choice alias="F" value="Female" />
      <Choice alias="NB" value="Nonbinary" />
      <Choice alias="X" value="Other" />
    </Choices>
  </View>
</View>

Presumably, this is a bug?

[Gondragos]

Oh, I'm afraid that the version of label-studio-frontend library on the playground page is outdated. It's better to check it directly in Label Studio. I just have tested it and it works for me well.

[asymingt]

OK, whoops! That's my go-to for trying things out. I've always assumed it's kept up to date with the latest release. Thank you for confirming. I will check on my local installation to confirm. Otherwise, I think this feature request is probably already covered by existing functionality. Thank you so much for taking the time to look into it and help me out!