search

sachit-menon / classify_by_description_release

160 stars 24 forks source link

strange result for Oxford Pets dataset #10

Closed guolipin closed 4 months ago

guolipin commented 4 months ago

Hi, sachit. May I have your help for the output of the pet dataset?

Model = RN50 | Datasets: pets Total Description-based Top-1 Accuracy: 11.092940717935562 Total CLIP-Standard Top-1 Accuracy: 10.41155606508255

Model = ViT-L/14 | Datasets: pets Total Description-based Top-1 Accuracy: 10.520578175783157 Total CLIP-Standard Top-1 Accuracy: 11.229217797517776

This was extremely low accuracy compared to the published outcome in the research paper. I didn't modify the code, but I got such result.

guolipin commented 4 months ago

Oh, I fixed it, the order of keys in the json file is not matching the order as the dataset.classes. Need to change the order in the json file