Binary Segmentation

[Troedelmaus]

Hello,

I'd like to segment a single object in a picture (only two Instances being recognized -> Foreground and Background). I've tried tweaking the parameters in "SamAutomaticMaskGenerator" but failed to get satisfying results. Any leads?

[franchesoni]

To segment an object you should do a click. The unsupervised segmentator uses clicks in a grid and then post-processes the outputs. One thing you could do is to use a classifier for your binary class over the image, compute the attention or gradcam or whatever score you prefer to give importance to the pixels, and then click on the pixel with the largest score.

[AkshitSharma1]

To segment an object you should do a click. The unsupervised segmentator uses clicks in a grid and then post-processes the outputs. One thing you could do is to use a classifier for your binary class over the image, compute the attention or gradcam or whatever score you prefer to give importance to the pixels, and then click on the pixel with the largest score.

@franchesoni Sir I am not exactly clear with the concept of clicks, what are they and how exactly are they being calculated? Please could you explain a bit further, I am myself trying to predict a foreground/background mask for cell images. Any leads would be greatly helpful sir.

[AkshitSharma1]

@franchesoni I am actually trying to use SAM to further refine the mask produced by another model (thereby increasing accuracy). I have tried various approaches mentioned in another github thread but have recieved no success so far, is there any way I may contact you to please discuss the same further?

[hongminpark]

To segment an object you should do a click. The unsupervised segmentator uses clicks in a grid and then post-processes the outputs. One thing you could do is to use a classifier for your binary class over the image, compute the attention or gradcam or whatever score you prefer to give importance to the pixels, and then click on the pixel with the largest score.

@franchesoni Sir I am not exactly clear with the concept of clicks, what are they and how exactly are they being calculated? Please could you explain a bit further, I am myself trying to predict a foreground/background mask for cell images. Any leads would be greatly helpful sir.

@AkshitSharma1 'click' in here means user input prompt. In perspective of UI, user clicks exact point of object. You should use using SAM with prompts to do background removal through SAM(I guess.). \I am trying to use SAM to detect one object like you :)

[happyeungin]

@hongminpark @AkshitSharma1 do you guys find out the answer how to do this? I am stuck too

[hongminpark]

@happyeungin Couldn't find with SAM. SAM is too heavy for background removal :( But there are many other solutions.

• BackgroundRemoval library : This is python library and it's the easiest one for me. Those 2 are the most famous.
  • https://github.com/danielgatis/rembg
  • https://github.com/nadermx/backgroundremover
• Yolo segmentation : Fine tune with yolo model, you can train to detect and segment. Refer like this : https://blog.roboflow.com/train-yolov7-instance-segmentation-on-custom-data/
• Or you can give prompt imput and get segmented result. Refer this repositories https://github.com/IDEA-Research/Grounded-Segment-Anything

But all of the solutions had good and bad each. BackgroundRemoval is easy but not accurate than SAM, SAM is heavy, GroundSAM needs prompt inputs and else. I week ago, Faster SAM https://www.youtube.com/watch?v=zUweGzOGKQw was released, you check it out too. (I didn't yet)

[happyeungin]

@hongminpark Thanks, I am currently using rembg with DIS. It is working quite well generally, but sometimes not great.

It doesn't work when the original image has a shadow. After background removal, the shadow will still stay. So I was thinking If I should use object detection after removing the background to get good results.

Thanks for your suggestion, I will look into yolo and faster/grounded sam

[hongminpark]

@happyeungin (She's so cute) I encountered the same issue when using semantic segmentation(which is used in rembg). So I tried like this : Before running background removal, apply simple filters which adjust brightness/contrast/shadows to remove shadow. If you can recognize shadow is removed in your eyes, run remove bg and you'll get img without shadow. I just did like this.

Hope you find good solution!

[happyeungin]

@hongminpark Do you have any recommended filters to remove shadows? I tried Unsharp() but it just makes the shadow even more heavy.

[happyeungin]

She is very cute (but I am starting to forget she is in the image) as I have been staring at the shadow more than her for at least 2 weeks haha still trying to figure out