Integrating SAM in label studio

[jpodivin]

SAM[0] has been released couple of weeks ago. It's a model from Meta, capable of zero shot image segmentation on per-pixel basis and was released under Apache 2.0 license. The dataset was also released under Apache 2.0. Several versions of the model are available for download, and can be run locally using only cpu. Repo[1] of the project provides simple guide to using the model on images, producing bounding boxes and masks for objects.

The utility seems obvious. If this could be integrated with label-studio it would be possible to pre-annotate images en-masse, before submitting them to humans for labeling. Instead of drawing bounding boxes, the labelers would only assign classes to bounding boxes produced by the model.

Introducing this as a feature would substantially simplify creation of new image datasets.

[0]https://segment-anything.com/ [1]https://github.com/facebookresearch/segment-anything

[shondle]

Hey, I just created a pull request for an implementation on the label-studio-ml-backend github here.

If you want to go straight to the code, you can view it here

Hope this helps! :)

[lukasugar]

Hi, any ETA on when this feature will be available in production?

[makseq]

@lukasugar we are reviewing and testing it. It would be valuable if you could test it from the shondle's PR too and leave a feedback.

[debpalash]

Most Export Methods are not available. How to fix this?

<View>
  <Image name="image" value="$image" zoom="true"/>
  <BrushLabels name="tag" toName="image">
    <Label value="bicycles" background="#FF0000"/>
    <Label value="boats" background="#FF0000"/>
    <Label value="bridges" background="#FF0000"/>
    <Label value="buses" background="#0d14d3"/>
    <Label value="chimneys" background="#0d14d3"/>
    <Label value="crosswalks" background="#0d14d3"/>
    <Label value="fire hydrants" background="#0cc0ed"/>
    <Label value="motorcycles" background="#0d14d3"/>
    <Label value="parking meters" background="#0d14d3"/>
    <Label value="stairs" background="#0d14d3"/>
    <Label value="taxis" background="#0d14d3"/>
    <Label value="tractors" background="#0d14d3"/>
    <Label value="traffic lights" background="#0d14d3"/>
    <Label value="vehicles" background="#0d14d3"/>
  </BrushLabels>
  <KeyPointLabels name="tag2" toName="image">
    <Label value="bicycles" smart="true" background="#bb5fec" showInline="true"/>
    <Label value="boats" smart="true" background="#506dfb" showInline="true"/>
    <Label value="bridges" smart="true" background="#c42727" showInline="true"/>
    <Label value="buses" smart="true" background="#32ec79" showInline="true"/>
    <Label value="chimneys" smart="true" background="#109d6e" showInline="true"/>
    <Label value="crosswalks" smart="true" background="#90ce1c" showInline="true"/>
    <Label value="fire hydrants" smart="true" background="#0cc0ed" showInline="true"/>
    <Label value="motorcycles" smart="true" background="#000000" showInline="true"/>
    <Label value="parking meters" smart="true" background="#000000" showInline="true"/>
    <Label value="stairs" smart="true" background="#000000" showInline="true"/>
    <Label value="taxis" smart="true" background="#000000" showInline="true"/>
    <Label value="tractors" smart="true" background="#000000" showInline="true"/>
    <Label value="traffic lights" smart="true" background="#000000" showInline="true"/>
    <Label value="vehicles" smart="true" background="#000000" showInline="true"/>

    <Label value="Bus Eraser" smart="true" background="#000000" showInline="true"/>
  </KeyPointLabels>
</View>

[erinmikailstaples]

Hey @jpodivin + @lukasugar — thanks so much for opening this up + reaching out to close the loop here! Thanks to @shondle's help, we now support SAM within Label Studio.

Check out more about the integration in our integration directory: https://labelstud.io/integrations/machine-learning/segment-anything-model/

On our blog: https://labelstud.io/blog/exploring-the-powerful-segment-anything-model-integration/

Or on YouTube: https://www.youtube.com/watch?v=mUnvYTZdShk

Thanks so much for reaching out on this!

[erinmikailstaples]

And @debpalash — thanks so much for posting your note, are you able to create a new issue with further detail about your setup, integration, and context?

for clarity and to keep this organized, I'm closing this issue in the meantime. :)

[spunknic]

Following the steps from https://labelstud.io/blog/get-started-using-segment-anything/ After executing docker build . -t sam:latest, the process fails with the following error message: ....

[+] Building 2.2s (11/19) docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.73kB 0.0s => resolve image config for docker.io/docker/dockerfile:1 0.7s => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50 0.0s => [internal] load metadata for docker.io/library/python:3.8-slim 0.5s => [internal] load .dockerignore 0.0s => => transferring context: 216B 0.0s => [python-base 1/13] FROM docker.io/library/python:3.8-slim@sha256:72ae14e80c21f274f31111debd505d8fa64536fdf41 0.0s => [internal] load build context 0.0s => => transferring context: 25.87kB 0.0s => CACHED [python-base 2/13] WORKDIR /app 0.0s => CACHED [python-base 3/13] RUN --mount=type=cache,target="/var/cache/apt",sharing=locked --mount=type=cac 0.0s => CACHED [python-base 4/13] COPY download_models.sh . 0.0s => ERROR [python-base 5/13] RUN bash /app/download_models.sh 0.5s ------ > [python-base 5/13] RUN bash /app/download_models.sh: 0.500 /app/download_models.sh: line 2: $'\r': command not found 0.504 /app/download_models.sh: line 5: $'\r': command not found 0.505 /app/download_models.sh: line 6: syntax error near unexpected token `$'{\r'' '.505 /app/download_models.sh: line 6: `download_model() { ------ Dockerfile:31

[sv-hmelevsky]

Following the steps from https://labelstud.io/blog/get-started-using-segment-anything/ After executing docker build . -t sam:latest, the process fails with the following error message: ...

[segment_anything_model python-base 5/13] RUN bash /app/download_models.sh: 0.229 /app/download_models.sh: line 2: $'\r': command not found 0.230 /app/download_models.sh: line 5: $'\r': command not found 0.230 /app/download_models.sh: line 6: syntax error near unexpected token $'{\r'' '.230 /app/download_models.sh: line 6:download_model() {

I alsow recive same error too! In my case change enconding from CRLF to LF at download_models.sh file is helped me and solved the problem. Gl!