Open ZichengDuan opened 12 months ago
I also tried dot product (cosine similarity) between the output class token embeddings, and the similarity scores are also random. In case of implementation error, I tried 3 implementations (from knn.py, from PyTorch, from my own implementation) and all of them consistently returned the same result:
Pseudo Input/Output:
CASE 1:
In: img 1 = "Cat01.jpg"
Out img2 = "Cat01.jpg"
cosine similarity: 1
CASE 2:
In: img 1 = "Cat01.jpg"
Out img2 = "Dog01.jpg"
cosine similarity: 0.0227
CASE 3:
In: img 1 = "Cat01.jpg"
Out img2 = "Cat02.jpg"
cosine similarity: -0.0247
Below are the images I used:
Cat01.jpg
Dog01.jpg
https://github.com/facebookresearch/dinov2/blob/main/dinov2/data/transforms.py#L42 i guess your transform function is overloaded by the second one, which uses a wrong mean and variance value
transform = T.Compose([
T.Resize(224),
T.CenterCrop(224),
T.ToTensor(),
T.Normalize(mean=[0.5], std=[0.5]),
])
img1 = transform(img1)[:3].unsqueeze(0)
img2 = transform(img2)[:3].unsqueeze(0)
https://github.com/facebookresearch/dinov2/blob/main/dinov2/data/transforms.py#L42 i guess your transform function is overloaded by the second one, which uses a wrong mean and variance value
transform = T.Compose([ T.Resize(224), T.CenterCrop(224), T.ToTensor(), T.Normalize(mean=[0.5], std=[0.5]), ]) img1 = transform(img1)[:3].unsqueeze(0) img2 = transform(img2)[:3].unsqueeze(0)
Sorry for causing such a misunderstanding, I deleted this transform function in my own code and I simply forgot to delete it here. Thanks for mentioning this, I have edited the code above. :)
Did you get a chance to try a simple L2 distance between the [CLS] tokens ? This would be close to what we use with the knn classifier.
Hi, I see that cross-entropy loss is introduced in the paper as a metric to calculate the similarity between teacher results and student results, I wonder how to extend DINOv2 to the calculation of similarity between two different images.
With such cross-entropy loss supervision, in my understanding, the model learns a general representation of objects in the same categories, can I assume that the network can determine the similarity between two different images by calculating the cross-entropy? I tried to calculate the cross-entropy between different image pairs but the results seem random.
And here's my code (please ignore my dummy coding style :) ):
Does anyone have any insights about these cases? Am I on the right track?