Open GenevieveBuckley opened 11 months ago
Thanks for the pointer! We're aware of that one already and it's on the list for looking into faster SAM implementations ;).
I just went ahead with the vit_t
in #174 because it is so easy to integrate.
I changed the title to indicate the broader scope. As mentioned, a student will start to work on this mid of October, and @anwai98 and me are already working on better benchmarks for the models (that reflect interactive segmentation quality). See this doc for an overview of our (preliminary) plans for evaluating the faster models.
Just came across another potentially interesting project: https://github.com/SysCV/sam-hq
I haven't looked at this yet, but it might be worth figuring out if it could be useful for microscopy datasets (which are often high resolution images with a lot of fine-grained detail).
Another candidate: https://arxiv.org/pdf/2312.13789.pdf (similar to MobileSAM, but claims better performance).
I tried TinySAM (the method from my previous reply), but it does not work for our data; the bounding box prompt is somehow broken.
However, we now have MobileSAM, which also uses a vit_tiny backbone, integrated nicely. Will summarize this soon.
Just as a follow up here:
Based on this I think the best strategy is:
vit_l
as the default model. (It's faster than vit_h
without any penalty in performance, or even slightly better performance).For the next release we will go with the strategy outlined above, so I removed this issue from the milestone. Will leave it open though for future discussions.
I know there are a whole bunch of projects that popped up after segment-anything was released, but another one we might like to look at and potentially include is FastSAM: https://github.com/CASIA-IVA-Lab/FastSAM
The model checkpoints are linked from the README here: https://github.com/CASIA-IVA-Lab/FastSAM#model-checkpoints
Cross-reference: https://github.com/computational-cell-analytics/micro-sam/pull/174