search

HumanSignal / label-studio

Label Studio is a multi-type data labeling and annotation tool with standardized output format
https://labelstud.io
Apache License 2.0
19.5k stars 2.42k forks source link

Smart Polygon and Brush don't exist despite mention in docs #6221

Closed jneeven closed 2 months ago

jneeven commented 3 months ago

Describe the bug The docs of https://labelstud.io/tags/polygon describe the use of the smart and smartOnly tags, as with most other labels. The ML labeling guideline mentions the Polygon and Brush as well: https://labelstud.io/guide/ml.html#Smart-tools

However, I find that while Rectangle and Keypoint work as expected, Polygon and Brush do not work at all. I have not tested the Ellipse. For the smart brush no web request is sent to the backend, and for the smart polygon no button even appears. Why do these not work? Are the docs incorrect or am I doing something wrong?

To Reproduce Set up a new project with the following config:

<View>
  <Header value="Select label and click the image to start"/>
  <Image name="image" value="$image" zoom="true"/>
  <PolygonLabels name="label" toName="image"
                 strokeWidth="3" pointSize="small"
                 opacity="0.9" smart="true">
    <Label value="Airplane" smart="true" background="red"/>
    <Label value="Car" smart="true" background="blue"/>
  </PolygonLabels>
</View>

or alternatively, since the docs only mention the smart tag on the Polygon and not PolygonLabels:

<View>
  <Header value="Select label and click the image to start"/>
  <Image name="image" value="$image" zoom="true"/>
  <Polygon name="label" toName="image"
                 strokeWidth="3" pointSize="small"
                 opacity="0.9" smart="true"/>
</View>

Then specify an ML backend, e.g. SAM2-Image, and enable interactive annotations

image

Then import a sample image and open the label interface. image

You see that there is no icon for smart polygon despite auto-annotation being enabled. Looking at the logs of the ML backend, no request is received upon drawing a regular polygon either.

Expected behavior I would expect the smart polygon to send a request to the ML backend on each update, so that I can process it with an ML model and return an appropriate prediction.

Smart keypoints and boxes are already useful, but I find myself labeling "thin" shapes where a bbox is too large and I would require many keypoints to accurately capture all parts of the object... Polygons are a lot more useful (then I can create keypoints from the polygon myself and feed those to e.g. SAM2)

Environment (please complete the following information):

Thanks!

makseq commented 2 months ago

I don't think LS SAM2 ML Backend supports PolygonLabels:

It generates BrushLabels masks: https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/segment_anything_2_image/model.py#L109

And it waits for KeypointLabels or RectangeLabels as "magic" auto-annotation helpers: https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/segment_anything_2_image/model.py#L128

sven-knoblauch commented 2 months ago

I have the same problem with the smart polygon, where no icon is enabled and no request is send to the ml backend (using a custom ml-backend, which can use polygons). This feature will be discarded?

jneeven commented 2 months ago

@makseq that's a separate issue. The backend doesn't even receive any request, so regardless of whether the model supports it it won't work. SAM2 is just an example, in reality I'm running a custom model.