create_prediction result is None

[berombau]

I have 512x512 greyscale images I want to annotate using brush masks by humans and automatically segment later on using a UNet ML backend.

Versions: label-studio: 1.4 label-studio-sdk: 0.0.4 label-studio-converter: 0.0.36

I use the Python SDK and want to create image predictions similar to this example: https://github.com/heartexlabs/label-studio-ml-backend/blob/feature/example-image-segmentation/label_studio_ml/examples/interactive_segmentation/interactive_segmentation.py Although my images are greyscale, I assume the mask will also be RGBA as in the example. This example code should generate a ~50% random mask for the Mitochondria class over the image, but it generates a buggy prediction instead:

import numpy as np
import label_studio_converter.brush as brush
from label_studio_sdk import Client

LABEL_STUDIO_URL = 'http://localhost:8080'
LABEL_STUDIO_API_KEY = '92263ad5544e90d0c730aa088187994ba4cc3626'
LABEL = 'Mitochondria'

ls = Client(url=LABEL_STUDIO_URL, api_key=LABEL_STUDIO_API_KEY)
ls.check_connection()

project = ls.start_project(
    title=LABEL,
    label_config=f"""
    <View>
    <Image name="image" value="$image" zoom="true"/>
    <BrushLabels name="tag" toName="image">
        <Label value="{LABEL}" background="#8ff0a4"/>
    </BrushLabels>
    </View>
    """
)

ids = project.import_tasks([
    {'image': f'http://localhost:7894/data_{i:04}.png'} for i in range(64)
])

result_mask = np.random.randint(low=0, high=256, size=(512, 512))
result_mask[result_mask > 100] = 255
result_mask = result_mask.astype(np.uint8)
# convert mask to RGBA image
import PIL
got_image = PIL.Image.fromarray(result_mask)
rgbimg = PIL.Image.new("RGBA", got_image.size)
rgbimg.paste(got_image)

datas = rgbimg.getdata()
# make pixels transparent
newData = []
for item in datas:
    if item[0] == 0 and item[1] == 0 and item[2] == 0:
        newData.append((0, 0, 0, 0))
    else:
        newData.append(item)
rgbimg.putdata(newData)
# get pixels from image
pix = np.array(rgbimg)
# rgbimg.save("test.png")
# encode to rle
result_mask = brush.encode_rle(pix.flatten())
result_mask

project.create_prediction(task_id=ids[0], model_version='123', result = {
    "original_width": 512,
    "original_height": 512,
    "image_rotation": 0,
    "from_name": "tag",
    "to_name": "image",
    "type": "brushlabels",
    'value': {
        "format": "rle",
        "rle": result_mask,
        "brushlabels": [LABEL]
    }
})

The response from create_prediction shows None for the result:

{'id': 238,
 'model_version': '123',
 'created_ago': '0\xa0minutes',
 'result': None,
 'score': 0.0,
 'cluster': None,
 'neighbors': None,
 'mislabeling': 0.0,
 'created_at': '2021-12-10T12:12:11.375195Z',
 'updated_at': '2021-12-10T12:12:11.375247Z',
 'task': 641}

In the Label Studio UI, the prediction is created for the task, but result is also null when looking at the task source code. The page for the task itself does not load and does not show other annotations, only the single buggy prediction.

There are no error messages in the label-studio log to further debug this problem. Is this maybe not the correct way to generate an image segmentation prediction?

[berombau]

I solved it by looking again at interactive_segmentation.py and the function brush.image2annotation. The result object is not a dict, but a list of dicts for image segmentation. This is wrongly documented at https://labelstud.io/tags/brushlabels.html and fails silently. The expected behavior for a Result object of type brushlabels should be either throwing an error if it is not a list or the dict should be converted to a list of length 1 so it is inline with the other Result object types for single class segmentation use cases.

Here is the working example with some simplifications based on https://labelstud.io/guide/predictions.html#Import-brush-segmentation-pre-annotations-in-RLE-format:

import numpy as np
import label_studio_converter.brush as brush
from label_studio_sdk import Client

LABEL_STUDIO_URL = 'http://localhost:8080'
LABEL_STUDIO_API_KEY = '92263ad5544e90d0c730aa088187994ba4cc3626'
LABEL = 'Mitochondria'

ls = Client(url=LABEL_STUDIO_URL, api_key=LABEL_STUDIO_API_KEY)
ls.check_connection()

project = ls.start_project(
    title=LABEL,
    label_config=f"""
    <View>
    <Image name="image" value="$image" zoom="true"/>
    <BrushLabels name="tag" toName="image">
        <Label value="{LABEL}" background="#8ff0a4"/>
    </BrushLabels>
    </View>
    """
)

ids = project.import_tasks([
    {'image': f'http://localhost:7894/data_{i:04}.png'} for i in range(64)
])

mask = (np.random.random([512, 512]) * 255).astype(np.uint8)  # just a random 2D mask
mask = (mask > 128).astype(np.uint8) * 255  # better to threshold, it reduces output annotation size
rle = brush.mask2rle(mask)  # mask image in RLE format

project.create_prediction(task_id=ids[0], model_version=None, result = [{
    "from_name": "tag",
    "to_name": "image",
    "type": "brushlabels",
    'value': {
        "format": "rle",
        "rle": rle,
        "brushlabels": [LABEL]
    }
}])

[makseq]

This definitely deserves to be posted in SDK: https://github.com/heartexlabs/label-studio-sdk/blob/master/examples/import_preannotations/import_brush_predictions.py Thank you!