How to get subpart/part/whole-level segmentation like SAM？

[chenkangjie1123]

Hello, I would like to know: the original SAM model can output segmentation results at three levels: whole, part, and subpart. Does your method now support such output? Because I tried to modify the two parameters --suppress_small_objects --SAM_PRED_IOU_THRESHOLD and found that The result is only that there is a difference in the range of the segmented areas, but it seems that no difference in scales such as subpart, part, and whole is seen?

[hkchengrex]

You can modify the segment selection process of SAM in this class: https://github.com/hkchengrex/Tracking-Anything-with-DEVA/blob/752d2f5e65a7d5efdd55e55f48932678798ef3d8/deva/ext/SAM/automatic_mask_generator.py#L41

We do find these granularities to be temporally highly unstable.

[chenkangjie1123]

Indeed, I have modified this part of the code and found that when deva performs mask matching on the whole scale, some frame images may completely lose the mask, while the subpart and part scale results are okay.

[HuangWenH]

@hkchengrex @chenkangjie1123 This is a great work. Could you tell me which parameters to adjust to achieve multi-scale cutting? I tried to modify the crop_n_layers parameter, but KeyError: 'crop_boxes error occurred. How can I adjust the parameters to achieve smaller scale cutting, such as cutting car tires and other details?

[hkchengrex]

@HuangWenH I don't think there are explicit definitions of those granularities in SAM. It was just an observation. So we would have to extract/compare those scales manually with post-processing. This can be done within the SAMAutomaticMaskGenerator, more specifically here: https://github.com/hkchengrex/Tracking-Anything-with-DEVA/blob/752d2f5e65a7d5efdd55e55f48932678798ef3d8/deva/ext/SAM/automatic_mask_generator.py#L317-L329