Closed jaiswati closed 2 years ago
First - for DINO you can just use the self attention from the last layer, and visualize it as a 2D image.
Another option, using this repo, since you don't have categories, would be to use the "EigenCAM" method. This will find salient objects in the feature representations.
And yet another option - A few minutes ago I added a notebook tutorial for visualizing concept embeddings in images: https://github.com/jacobgil/pytorch-grad-cam/blob/master/tutorials/Pixel%20Attribution%20for%20embeddings.ipynb
It works on models that output embedding feature vectors (like DINO or other SSL models), and searches for concept embeddings in the image. In SSL you would have to define these concepts - they could be samples of the training images, for example.
Hello @jacobgil
Thanks a lot for these awesome repositories :) is it possible to access the ViT backbone of pretrained DINO network and then explain it with grad-cam methods? Probably accessing specific layers etc. kindly help with how exactly it could be done? Best Regards, @jaiswati