Predictions posted but not showing using ml backend

[IT-Guy007]

Describe the bug Using the combination of Label studio and Label studio LM doesn't seem to show predictions.

Using LS for OCR and correction with the following template:

<View>
  <Image name="image" value="$ocr"/>

  <Labels name="label" toName="image">
    <Label value="Key" background="green"/>
    <Label value="Value" background="blue"/>
  </Labels>

  <Rectangle name="bbox" 
             toName="image" 
             strokeWidth="3"
             smart="true"
             />
  <Polygon name="poly" 
           toName="image" 
           strokeWidth="3"
           smart="true"
           />

  <TextArea name="transcription" 
            toName="image"
            editable="true" 
            perRegion="true" 
            required="true" 
            maxSubmissions="1" 
            rows="5" 
            placeholder="Recognized Text" 
            displayMode="region-list"
            />
</View>

LS sends a request to the LS-ML container it processes it and sends back the result as shown below. The response from lm: [2023-11-06 11:13:08,626] [INFO] [werkzeug::_log::96] 172.19.0.3 - - [06/Nov/2023 11:13:08] "POST /predict HTTP/1.1" 200 -

{
    "results": [
        {
            "data": {
                "ocr": "/data/upload/1/6cdaa317-Capture.PNG"
            },
            "predictions": [
                {
                    "model_version": "0.0.1",
                    "result": [
                        {
                            "from_name": "bbox",
                            "id": "17acaa68-3",
                            "image_rotation": 0,
                            "original_height": 89,
                            "original_width": 206,
                            "to_name": "image",
                            "type": "rectangle",
                            "value": {
                                "height": 10.112359550561798,
                                "rotation": 0,
                                "width": 12.135922330097088,
                                "x": 11.650485436893204,
                                "y": 24.719101123595507
                            }
                        },
                        {
                            "from_name": "label",
                            "id": "17acaa68-3",
                            "image_rotation": 0,
                            "original_height": 89,
                            "original_width": 206,
                            "to_name": "image",
                            "type": "labels",
                            "value": {
                                "height": 10.112359550561798,
                                "labels": [
                                    "value"
                                ],
                                "rotation": 0,
                                "width": 12.135922330097088,
                                "x": 11.650485436893204,
                                "y": 24.719101123595507
                            }
                        },
                        {
                            "from_name": "transcription",
                            "id": "17acaa68-3",
                            "image_rotation": 0,
                            "original_height": 89,
                            "original_width": 206,
                            "to_name": "image",
                            "type": "textarea",
                            "value": {
                                "height": 10.112359550561798,
                                "rotation": 0,
                                "text": [
                                    "First name: John \\u2018Surname: Doe"
                                ],
                                "width": 12.135922330097088,
                                "x": 11.650485436893204,
                                "y": 24.719101123595507
                            }
                        }
                    ],
                    "score": 0.8956609216666666
                }
            ]
        }
    ]
}

But when viewing it in the LS the results don't appear.

To Reproduce Steps to reproduce the behavior:

1. Click on retrieve predictions

Expected behavior Expected to see the bounding boxes to appear

Environment (please complete the following information):

• OS: Docker :latest and :develop tried
• Label Studio Version 1.9.1.post0(updated using Dockerfile)

Additional context

• In the machine learning section of the project and then the only backend, under version it says in red letters "Upgrade your ML backend version to latest"
• Under actions: create annotations from predictions, there isn't any model selectable

[hogepodge]

@IT-Guy007 can you share a bit more information with us? In particular what does the ML Connection screen look like.

Label Studio will only show predictions for the model version specified in that control panel, and if you've the fit method you may be cycling through model versions, hiding the predictions from the labeling interface.

[ScrambledRK]

Not sure, but ... I also use the following fields for result objects.

"source": "$image",
'from_name': "label",
'to_name': "image",

The values "label" and "image" seem to correspond with the label configuration (checkout the label configuration XML of your project). Maybe even "$image" for source, I cannot remember.

It works for me, but I hardly know why.

[IT-Guy007]

Hi,

See the attachment for the ML page, In some issues I saw that the /setup api call needs to be done. I cant find any documentation about it

When I get the health from the ml it says:

{
  "model_class":"tesseractModel",
  "status":"UP"
}

When clicking on "start training":

{
  "model_version":"INITIAL"
}

I dont have a fit class, furthermore have I tried adding the key source to the dict, but that doesnt work either. It's something with the setup of the ml.

My model.py:

from typing import List, Dict, Optional
from label_studio_ml.model import LabelStudioMLBase

from tesseract import tessaractClass

class tesseractModel(LabelStudioMLBase):

    def predict(self, tasks: List[Dict], context: Optional[Dict] = None, **kwargs):

        results = []

        for task in tasks:
            results.append(tessaractClass.predict(task))

        return results

    def fit(self, event, data, **kwargs):
        print("Fit called")

[IT-Guy007]

hi @hogepodge, @ScrambledRK,

Do you guys have any idea? I can't remove the label community reviewed. So this issue will be unwantengly ignored.

[ScrambledRK]

You are very sparse with your description - I guess your problem is that it says "INITIAL"? ... or are the predictions still not showing on your image? ... what about a "fit class"? what do you mean? what are you asking?

In case its about the version you may need to set self.set("model_version", version)in your Model somewhere. But it just seems quite buggy regardless. I don't have that working either.

[hogepodge]

If you're still running into these issues, I strongly suggest you unselect "allow version auto-update" unless you've implemented the fit() method and want to do active learning. When you're working in your labeling interface, you will need to explicitly set which model predictions you want to see.

[BMukhtar]

FYI: using i was using integer as id for resultId and it caused loading screen to halt endlessly, and just converting it to string solved my problem.

So some little problem in your json may cause endless loading while viewing predictions

[BMukhtar]

another solution can be using camelcase 'Value' instead of 'value':