search

Junyi42 / GeoAware-SC

Official Implementation of paper "Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence"
https://telling-left-from-right.github.io
97 stars 8 forks source link

Processing Features with Masks #13

Closed itaouil closed 3 weeks ago

itaouil commented 3 weeks ago

Hi,

Thanks a lot for the amazing work and for making it available to the community.

Is there the possibility to find matching features given two images using masks extracted from SAM2 (i.e. I do not pass the RGB image, but an RGB image masked in order to concentrate only on the object of interest). How would you recommend proceeding for something like this?

Also, are there specific parameters that you would recommend tweaking to obtain the best feature matching performance?

Best, Ilyass

Junyi42 commented 3 weeks ago

Hi @itaouil,

Thanks for your interest in our work!

Honestly, I haven't tried images that are masked for extracting the features, and I am unsure whether there will be a domain shift issue. Another way is to run the feature extractor on the whole feature map and use the SAM2 mask to index the features of interest (you could also try to use bilinear upsampling or FeatUp on the feature map before applying the mask).

Please feel free to let me know if you have further questions. Thanks!

Best.

itaouil commented 3 weeks ago

Hi @Junyi42,

Thanks for your quick reply.

So you would suggest to run the feature extractor on the whole images, without masking or cropping the image around the object of interests that I would like to match (in this case imagine I want to match two chairs in two different images) and use SAM2 to make sure the matched features are within the source and target image mask right?

Best, Ilyass

Junyi42 commented 3 weeks ago

Hi @Junyi42,

Thanks for your quick reply.

So you would suggest to run the feature extractor on the whole images, without masking or cropping the image around the object of interests that I would like to match (in this case imagine I want to match two chairs in two different images) and use SAM2 to make sure the matched features are within the source and target image mask right?

Best, Ilyass

Yes. :-)