search

aim-uofa / Matcher

[ICLR'24] Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
https://arxiv.org/abs/2305.13310
Other
448 stars 26 forks source link

Questions about the model task: object non-presence, multi-categories and few-shot #7

Open theodu opened 1 year ago

theodu commented 1 year ago

Hello, Congratulation for your great and interesting work! I have several questions to see if this model match my use case:

1- If I run the model on a target image without the reference object, will it still predict something or will it be able to say (with a given confidence) that the image does not have the queried object ? 2- I am interested to run this model with several categories as inputs. Is there a mecanism to run the inference on several categories at the same time or will I have to run distinct predictions for each categories ? 3- Can the model be extended to do few-shot with several reference masks for one same object ?

Thank you in advance!

yangliu96 commented 1 year ago

Hi, thank you for your attention!

  1. In theory, for a target image without a reference object, it should not detect any unrelated objects. However, this is controlled by the confidence threshold. If the threshold is relaxed, the target image may find objects that are most similar to the given reference object.
  2. Similar to handling VOS (detecting different objects simultaneously), if it can provide references for different categories simultaneously, Matcher has the potential to make predictions for different categories simultaneously.
  3. We recently conducted 5-shot experiments on COCO and LVIS. Matcher can be effectively extended to few-shot segmentation.
  COCO LVIS
1-shot 52.7 33.0
5-shot 60.7 40.0
theodu commented 1 year ago

That is great! Thank you very much for your answer! Are you able to give an approximate release date for the code ? Really looking forward to test it!