search

purnasai / Dino_V2

Dino V2 for Classification, PCA Visualization, Instance Retrival: https://arxiv.org/abs/2304.07193
135 stars 10 forks source link

How do you know the threshold when visualizing the feature? #4

Open luccachiang opened 3 months ago

luccachiang commented 3 months ago

Hi, your repo is really helpful. I have one minor question, how do you know this threshold pca_features_bg = pca_features[:, 0] > 0.35 in third-party/Dino_V2/2.PCA_visualization.ipynb IN [10]? I am not sure how to infer this number from the first histogram. Could you please explain it?

Moreover, according to the original paper of DINOv2, it says Background is removed by removing patches with a negative score of the first PCA component. (Figure 9 caption) May I ask what is the relationship between the negative score and your chosen threshold. Many thanks!

Varun-Tandon14 commented 3 months ago

Hi @Lucca-cherries. Although it might be clear to you for the sake of future visitors, I shall try to explain the answer in some detail. The value of 0.35 is the threshold chosen by the author for the min-max normalized values of the first PCA component. This value is in fact arbitrary and will depend totally upon your use case. For example please look at the histogram of the min-max normalized first PCA component below for two different scene conditions

1) Object kept in a well-lit room with no occlusions or other noise, i.e. very easy to differentiate between fg vs bg. PCA_1_fg_vs_bg 2) For another scene where the images are of a landmark taken from many different angles i.e. a little hard to differentiate between fg and bg. PCA_1_fg_vs_bg_noisy

From the above distributions, you can clearly see two peaks, which can be categorized as fg vs bg. Observing the above graphs, one can easily decide the correct threshold value for each scene.

Coming to your second point (although I haven't read the paper), I assume that the authors of the original papers either did not normalize the first PCA component or might have used some other normalization technique like mean etc to select only positive scored values as fg. So absolute threshold value in effect will also depend on your normalization technique. However, for the same set of images, the distribution should look the same.

Please visit this issue on the original repo for more details on this. Thanks for reading. And many thanks to the author for this awesome repo.