HumanSignal / label-studio

Label Studio is a multi-type data labeling and annotation tool with standardized output format
https://labelstud.io
Apache License 2.0
19.54k stars 2.42k forks source link

`origin` is not correctly set when predictions for `Choices` tag are modified #4046

Open minhduc0711 opened 1 year ago

minhduc0711 commented 1 year ago

Describe the bug After importing predicted data into Label Studio, a Choices prediction is modified. However, when exporting the annotations, the origin attribute has the value prediction instead of prediction-changed.

To Reproduce Steps to reproduce the behavior:

  1. Setup the following labeling UI
    <View>
      <Choices name="gender" toName="txt-1" choice="single-radio">
        <Choice alias="M" value="Male" />
        <Choice alias="F" value="Female" />
        <Choice alias="NB" value="Nonbinary" />
        <Choice alias="X" value="Other" />
      </Choices>
      <Text name="txt-1" value="$text" />
    </View>
  2. Import the following predicted data (Male is selected):

    [{
      "data": {
        "text": "Lorem ipsum bla bla." 
      },
    
      "predictions": [{
        "model_version": "one",
        "score": 0.5,
        "result": [
          {
            "id": "some_id",
            "type": "choices",
            "from_name": "gender", "to_name": "txt-1",
            "value": {
              "choices": ["M"]
          }
        }]
      }]
    }]
  3. Navigate to the labeling task, select Female and click Submit.
  4. Export the new annotation.

Expected behavior origin should have the value prediction-changed in the exported JSON. The relevant part in the actual exported JSON:

"result": [
    {
        "value": {
            "choices": [
                "F"
            ]
        },
        "id": "some_id",
        "from_name": "gender",
        "to_name": "txt-1",
        "type": "choices",
        "origin": "prediction"
    }
],
...
"prediction": {
    "id": 80662,
    "model_version": "one",
    "created_ago": "0 minutes",
    "result": [
        {
            "id": "some_id",
            "type": "choices",
            "from_name": "gender",
            "to_name": "txt-1",
            "value": {
                "choices": [
                    "M"
                ]
            }
        }
    ],

Environment (please complete the following information):

hlomzik commented 1 year ago

Currently this works only for labeling, not for classification. We'll file a bug and will fix it.

spagliarini commented 11 months ago

Hi! We are experiencing the same problem. Any update on the fix? @hlomzik