Difficulty in using Label Studios feature of using Pre-Annotated Images

[Yuvraj-Dhepe]

Describe the bug I am currently working with using SAM model, and trying to use the masks generated by SAM on multiple objects in image. After generation of masks, few objects are mis-masked, or not masked, thus I want to correct those masks inside the Label Studio.

As per Import Brush Segmentation Pre-Annotations in RLE Format, I tried converting the masked image to RLE Annotations, however after generating a tasks.json, file for an image and loading that into the Label Studio, The annotations are not loaded well.

To Reproduce Steps to reproduce the behavior: 1) Install SAM & Required Libraries pip install git+https://github.com/facebookresearch/segment-anything.git pip install git+https://github.com/facebookresearch/segment-anything.git

2) Load a random image of beans from the internet

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry
import cv2
import json
from PIL import Image
import os, sys
import fiftyone as fo
from label_studio_converter import brush

import numpy as np
import torch
import matplotlib.pyplot as plt
from utility_tools import image_utils as imz
import cv2

# Specify the path to the image you want to proce
image_path = "wbean.jpg"

# Open the image
image = Image.open(image_path)

# Resize the image
imResize = image.resize((512,512), Image.Resampling.LANCZOS)

# Get the filename and extension of the original image
f, e = os.path.splitext(image_path)

# Save the resized image
resized_path = './' + f"img_resized.jpg"
imResize.save(resized_path, "JPEG", quality=90)

# Loading the resized image
image = cv2.imread("img_resized.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

3) Make Predictions with SAM

# Downloading the weights of SAM
!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth

# Make Predictions
checkpoint = "sam_vit_l_0b3195.pth"
model_type = "vit_l"
device = "cuda"
sam = sam_model_registry[model_type](checkpoint=checkpoint)
sam.to(device=device)

mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=32,
    pred_iou_thresh=0.99,
    stability_score_thresh=0.97,
    crop_n_layers=2,
    crop_n_points_downscale_factor=2,
    min_mask_region_area=100,  # Requires open-cv to run post-processing
    output_mode = "binary_mask"
)
masks = mask_generator.generate(image)

4) Saving the colored masks to an image

def show_anns(anns):
    if len(anns) == 0:
        return
    sorted_anns = sorted(anns, key=(lambda x: x['area']), reverse=True)
    ax = plt.gca()
    ax.set_autoscale_on(False)

    img = np.ones((sorted_anns[0]['segmentation'].shape[0], sorted_anns[0]['segmentation'].shape[1], 4))
    img[:,:,3] = 0
    for ann in sorted_anns:
        m = ann['segmentation']
        color_mask = np.concatenate([np.random.random(3), [0.35]])
        img[m] = color_mask
    ax.imshow(img)

plt.figure(figsize=(10, 10))
plt.imshow(image)
show_anns(masks)
plt.axis('off')

# Save the figure with overlaid masks as a PNG image
plt.savefig('image_with_masks.png', bbox_inches='tight', pad_inches=0, dpi=300)

# Show the figure
plt.show() 

5) Convert the masks to RLE Format from the label studio Converter

path = "image_with_masks.png"
masks1 = cv2.imread(path)
#imz.display_images([masks1])
brush.image2annotation(path, label_name = 'Good', from_name='tag', to_name = 'image', ground_truth=False, model_version=None, score =None)

annotation = brush.image2annotation(
        path,
        label_name='good',
        from_name='tag',
        to_name='image',
        model_version='v1',
        score=0.5,
    )

# The path given here I got once I uploaded the image to label studio
task = {
        'data': {'image': '/data/upload/1/bfe92774-G12.png'},
        'predictions': [annotation],
    }

json.dump(task, open('task.json', 'w'))

6) I have installed label-studio locally, I run it on port 8080, after that, I simply create a new project with image segmentation template, once that's created, I simply upload the original image. After uploading the image, I take the path, and create a json, format code as mentioned above. N Finally upload the tasks.json file to Label Studio

Expected behavior I was expecting that I will be able to see all the beans with different colored masks, since SAM performed instance segmentation, however, I see all the beans uncolored inside the label studio as follows

Could you pls help me solve the issue, so that I see masks on the beans and correct the masks which are incorrectly generated by SAM

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: Ubuntu 22.04.3 LTS
• Label Studio Version Latest

[Yuvraj-Dhepe]

Hello @makseq, I was able to solve the issue, by using the SAM to generate masks, convert them to RLE format, by a mid way solution, of re-drawing contours on these masks, because for some reason, LABELSTUDIO doesn't recognize without re-drawing.

However, now once I have corrected the masks, I am onto finding a way how to convert those masks to yolov8 segmentation labels?

If you know a solution to that, it'll be really helpful.

Thank you

[connor-tan]

Hello @makseq, I was able to solve the issue, by using the SAM to generate masks, convert them to RLE format, by a mid way solution, of re-drawing contours on these masks, because for some reason, LABELSTUDIO doesn't recognize without re-drawing.

However, now once I have corrected the masks, I am onto finding a way how to convert those masks to yolov8 segmentation labels?

If you know a solution to that, it'll be really helpful.

Thank you

hello bro,have you been find out a way to solve this problem?

[Yuvraj-Dhepe]

@connor-tan I wasn't able to solve the problem. I just used detectron2 for my problem. It doesn't support semantic segmentation directly so I used the polygon based masks n trained the model for instance segmentation. But yah the performance was good enough for me.

[makseq]

@Yuvraj-Dhepe Did you add

 <Label value="good" background="rgba(255, 0, 0, 0.7)"/>

to your labeling config? Could you share the resulting tasks.json and the labeling config if you still have it?

[Yuvraj-Dhepe]

Sorry @makseq, unfortunately I don't have the config and tasks.json as of now. Maybe if I retry with your option, I will share the tasks.json or so.