search

microsoft / SoM

Set-of-Mark Prompting for LMMs
MIT License
1.08k stars 84 forks source link

Support additional segmentation models/parameters (e.g. NMS conf/iou thresholds) #43

Open abrichr opened 2 months ago

abrichr commented 2 months ago

Segmenting images with many distinct objects results in many objects being missed, e.g.:

image

(Notice not all cells are marked.)

In https://github.com/OpenAdaptAI/OpenAdapt/pull/679 we have good results using Ultralytics FastSAM for segmentation with very small values for non maximum suppression conf and iou:

    # threshold below which boxes will be filtered out
    conf: float = 0,
    # discards all overlapping boxes with IoU > iou_threshold
    iou: float = .05,

image

Incorporating FastSAM with parameterizable NMS would greatly increase the utility of SoM for tasks requiring high sensitivity and with many different objects.

Alternative suggestions for increasing the sensitivity with the current segmentation models would be greatly appreciated! 🙏

abrichr commented 2 months ago

We were able to get this working offline with FastSAM by just ripping out visualizer.py.