Result different from demo_vis_features.ipynb

[DevikalyanDas]

Hello @Junyi42 , Thanks for your contribution. I ran the "demo_vis_features.ipynb on the dog that was given in the default image folder. My results are coming different than yours. Yours masked pca result was

while I am getting

Also, my clustering is

I didn't change anything in the code only dumped everything from the ipynb to .py file and I am getting these outputs in the results_vis folder in the form of png files.

[Junyi42]

Hi,

I think this is mainly because the instance mask is not generated/loaded correctly, therefore, the background feature map is taken into account when computing the PCA and clustering.

You could refer to this #1 issue on how to generate the correct mask. Please feel free to let me know if you have any further issues.

Best, Junyi

[DevikalyanDas]

Hello, I added the parameter argument '''decoder= False''' in load_model and the result has improved and the clustering is looking like this

It was written that this mask is only for visualization purposes. So, this means if we need the features for some downstream tasks, we don't need the mask. Right? Also, you had mentioned in issue #1 that we can use SAM to obtain mask. So, the SAM mask will basically replace the get_mask function output?

[Junyi42]

Hi @DevikalyanDas ,

Sorry for the late reply.

As I could see, the background is still not masked (otherwise, it should be single colored). I would suggest using other segmentation methods to generate the mask.

Also, you had mentioned in issue https://github.com/Junyi42/sd-dino/issues/1 that we can use SAM to obtain mask. So, the SAM mask will basically replace the get_mask function output?

Yes, that's right. You could try to use different ways to get the instance mask.

So, this means if we need the features for some downstream tasks, we don't need the mask. Right?

That's right. The mask is only used for removing the background feature when visualizing the feature maps.

[DevikalyanDas]

Thanks, I got the output with single background similar to how you had provided in the jupyter notebook. Also, thanks for the explanation.