luca-medeiros / lang-segment-anything

SAM with text prompt
Apache License 2.0
1.42k stars 154 forks source link

Limited power of currently working text prompt driven segmentation #47

Open arv8315 opened 6 months ago

arv8315 commented 6 months ago

Instructions To Reproduce the 🐛 Bug:

  1. Background explanation

I have images of objects with some thin extursions (antennas). When I try to specify in the text prompt to obtain object with antennas attached to it, mostly prunes out the antennas. It is a sequence of images for the same object for which I need to subtract the background. Also, when it does get the antennas in very few images, the segmentation boundary is a bit imprecise and thin sliver of background is visible around antenna or parts of object where there are not as thin but somewhat thin extrusions. With some cavities that are naturally part of the object, bits of background visible through cavity leak in a bit.

I am going to attempt some post procesing clean up through contour detection if that works out to get a more crisp and better segmentation. Meanwhile is there a way to not have it cut out thin extrusions in the first place?

arv8315 commented 6 months ago

Workaround be as follows perhaps: Cut out part of the image and then use the text prompt to select the object of interest within that cutout and then merge it back to combine with the object where background is removed. Since we know where we made the cutting, we know where the position the component of interest is relative to the rest of the image. If we use this when combining all the segmentations together, may yield getting those extrusions also. I will try that and see how that performs.