search

facebookresearch / segment-anything

The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
Apache License 2.0
46.58k stars 5.52k forks source link

Trying to use SAM to segment engineering rock samples, but I need to solve two issues #749

Open citystrawman opened 4 months ago

citystrawman commented 4 months ago

Hello, I am trying to use SAM to segment engineering rock samples as follows: 企业微信截图_17152698856512 Here's a photo that contains a box of rock samples, and what I need to do is segment all the rocks and then choose the ones that is longer than a specific length(as we know the length of the box, we can easily calculate the length of each rock based on its pixel size).

By using SAM SamAutomaticMaskGenerator, I can get the segmented elements of this photo as follows: 企业微信截图_17152701264210

However, there're two issues which I think should be solved first so that my final target can be reached: The first one is the overlapping elements. As SAM may have overlapping segment elements, I may count repeated rocks. Is it possible to make SAM generate all non-overlapping elements? The second one is how to separate the rocks from backgrounds. SAM segments everything including photo background, and is it possible so that I can only segment rocks?

I appreciate if anyone could help solve this issue. Thank you !

mcihadarslanoglu commented 3 months ago

I think non-maximum overlapping suppression (NMS) can solve your overlapping problem an you can use a classifier to to distinguish which one is background or a rock. I am not sure of that works but you can also use a pretrained network to extract embeddings and calculate a similarity between all these objects. Since you have a lot of rock probably what you would like to segment will create a cluster.

citystrawman commented 3 months ago

I think non-maximum overlapping suppression (NMS) can solve your overlapping problem an you can use a classifier to to distinguish which one is background or a rock. I am not sure of that works but you can also use a pretrained network to extract embeddings and calculate a similarity between all these objects. Since you have a lot of rock probably what you would like to segment will create a cluster.

Thanks. As for non-maximum overlapping, I think segment-anything has already implemented it, but it still has possibility to generate overlapping masks. Just use the above images as an example, I have generated two images as follows: numbered top15

the first one lists all masks while the second one lists top 15 large masks. as you can see, mask 9 contains mask 58, mask 17, as well as mask 39. I think that is due to the semantic segmentation similar as the example shown by official: image