search

lukemelas / deep-spectral-segmentation

[CVPR 2022] Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization
227 stars 41 forks source link

Discussion: can the method differentiate the background classes? #11

Closed KevinChen880723 closed 1 year ago

KevinChen880723 commented 1 year ago

Hi, thank you for providing this awsome work!

After reading the paper, I realized that we are highly relying on color, spatial, and the features extracted by the DINO algorithm. So if we train a DINO model by the dataset that doesn't include background classes like road and sidewalk, and this two different classes shared the similar color and spatial features, can the eigen map still differentiate this two classes? Thanks again for the discussion!!

image

Kevin

lukemelas commented 1 year ago

Thanks for the question and for starting this discussion.

Yes, I agree that the model relies heavily on local features like texture and color. This is a property that has been observed in a bunch of other settings as well (for example, Deep ViT Features as Dense Visual Descriptors from Tali Dekel's group).

As to whether it can differentiate the classes, it depends heavily on the individual setting. Sometimes it is surprisingly good at separating classes, and other times it groups different classes together.

Hope this helps! Luke

KevinChen880723 commented 1 year ago

Thanks a lot for providing the good information and the discussion!

Kevin