Unable to reproduce the 1% and 10% NN results

ajtejankar commented 3 years ago

Hi,

I am interested in using the pretrained models, but I could not reproduce the numbers in the table. I downloaded the 1% checkpoints and ran the soft nearest neighbor evaluation on 1% ImageNet for all 3 checkpoints (100, 200, and 300 epochs). Here are the numbers that I got:

paws

Here's the command that I used:

for epoch in {100,200,300}
do
    python snn_eval.py \
        --model-name resnet50 \
        --use-pred \
        --pretrained weights/paws_imgnt_1percent_${epoch}ep.pth.tar \
        --unlabeled-frac 0.99 \
        --root-path /nfs/ada/hpirsiav/datasets \
        --image-folder imagenet \
        --dataset-name imagenet_fine_tune
done &> output/paws_imgnt_1percent_100_200_300_epochs

Here's the output of the above command:

+ for epoch in '{100,200,300}'
+ python snn_eval.py --model-name resnet50 --use-pred --pretrained weights/paws_imgnt_1percent_100ep.pth.tar --unlabeled-frac 0.99 --root-path /nfs/ada/hpirsiav/datasets --image-folder imagenet --dataset-name imagenet_fine_tune
INFO:root:loaded pretrained model with msg: <All keys matched successfully>
INFO:root:loaded pretrained encoder from epoch: 100 path: weights/paws_imgnt_1percent_100ep.pth.tar
INFO:root:making imagenet data transforms
INFO:root:keep file: imagenet_subsets/99percent.txt
INFO:root:data-path /nfs/ada/hpirsiav/datasets/imagenet/train/
INFO:root:Initialized ImageNet
INFO:root:ImageNet fine-tune dataset created
INFO:root:Using imagenet_subsets/99percent.txt
Namespace(dataset_name='imagenet_fine_tune', device='cuda:0', image_folder='imagenet', model_name='resnet50', normalize=True, pretrained='weights/paws_imgnt_1percent_100ep.pth.tar', root_path='/nfs/ada/hpirsiav/datasets', split_seed=152, subset_path='imagenet_subsets/', unlabeled_frac=0.99, use_pred=True)
weights/paws_imgnt_1percent_100ep.pth.tar
INFO:root:[0/801]
INFO:root:[50/801]
INFO:root:[100/801]
INFO:root:[150/801]
INFO:root:[200/801]
INFO:root:[250/801]
INFO:root:[300/801]
INFO:root:[350/801]
INFO:root:[400/801]
INFO:root:[450/801]
INFO:root:[500/801]
INFO:root:[550/801]
INFO:root:[600/801]
INFO:root:[650/801]
INFO:root:[700/801]
INFO:root:[750/801]
INFO:root:[800/801]
INFO:root:torch.Size([12811, 2048])
INFO:root:torch.Size([12811])
INFO:root:making imagenet data transforms
INFO:root:keep file: imagenet_subsets/0percent.txt
INFO:root:data-path /nfs/ada/hpirsiav/datasets/imagenet/val/
INFO:root:Initialized ImageNet
INFO:root:ImageNet fine-tune dataset created
INFO:root:flipping coin to keep labels
INFO:root:torch.Size([12811, 2048])
INFO:root:torch.Size([12811, 1000])
INFO:root:3125
INFO:root:[    0/3125] 56.250% 68.750%
INFO:root:[   50/3125] 60.172% 80.637%
INFO:root:[  100/3125] 61.510% 81.869%
INFO:root:[  150/3125] 61.714% 81.995%
INFO:root:[  200/3125] 61.318% 81.654%
INFO:root:[  250/3125] 61.479% 81.150%
INFO:root:[  300/3125] 61.047% 81.001%
INFO:root:[  350/3125] 61.556% 81.214%
INFO:root:[  400/3125] 61.300% 80.923%
INFO:root:[  450/3125] 61.183% 80.807%
INFO:root:[  500/3125] 61.190% 80.913%
INFO:root:[  550/3125] 60.923% 80.944%
INFO:root:[  600/3125] 61.023% 80.886%
INFO:root:[  650/3125] 61.089% 80.856%
INFO:root:[  700/3125] 61.198% 80.947%
INFO:root:[  750/3125] 61.293% 80.992%
INFO:root:[  800/3125] 61.236% 81.000%
INFO:root:[  850/3125] 61.259% 80.986%
INFO:root:Initialized ImageNet
INFO:root:ImageNet fine-tune dataset created
INFO:root:Using imagenet_subsets/99percent.txt
Namespace(dataset_name='imagenet_fine_tune', device='cuda:0', image_folder='imagenet', model_name='resnet50', normalize=True, pretrained='weights/paws_imgnt_1percent_200ep.pth.tar', root_path='/nfs/ada/hpirsiav/datasets', split_seed=152, subset_path='imagenet_subsets/', unlabeled_frac=0.99, use_pred=True)
weights/paws_imgnt_1percent_200ep.pth.tar
INFO:root:[0/801]
INFO:root:[50/801]
INFO:root:[100/801]
INFO:root:[150/801]
INFO:root:[200/801]
INFO:root:[250/801]
INFO:root:[300/801]
INFO:root:[350/801]
INFO:root:[400/801]
INFO:root:[450/801]
INFO:root:[500/801]
INFO:root:[550/801]
INFO:root:[600/801]
INFO:root:[650/801]
INFO:root:[700/801]
INFO:root:[750/801]
INFO:root:[800/801]
INFO:root:torch.Size([12811, 2048])
INFO:root:torch.Size([12811])
INFO:root:making imagenet data transforms
INFO:root:keep file: imagenet_subsets/0percent.txt
INFO:root:data-path /nfs/ada/hpirsiav/datasets/imagenet/val/
INFO:root:Initialized ImageNet
INFO:root:ImageNet fine-tune dataset created
INFO:root:flipping coin to keep labels
INFO:root:torch.Size([12811, 2048])
INFO:root:torch.Size([12811, 1000])
INFO:root:3125
INFO:root:[    0/3125] 56.250% 68.750%
INFO:root:[   50/3125] 62.868% 81.740%
INFO:root:[  100/3125] 63.057% 81.869%
INFO:root:[  150/3125] 63.411% 82.036%
INFO:root:[  200/3125] 62.935% 81.779%
INFO:root:[  250/3125] 62.998% 81.673%
INFO:root:[  300/3125] 62.708% 81.499%
INFO:root:[  350/312
INFO:root:[ 3050/3125] 63.123% 81.828%
INFO:root:[ 3100/3125] 63.137% 81.822%
INFO:root:top1/top5: 63.18/81.824
+ for epoch in '{100,200,300}'
+ python snn_eval.py --model-name resnet50 --use-pred --pretrained weights/paws_imgnt_1percent_300ep.pth.tar --unlabeled-frac 0.99 --root-path /nfs/ada/hpirsiav/datasets --image-folder imagenet --dataset-name imagenet_fine_tune
INFO:root:loaded pretrained model with msg: <All keys matched successfully>
INFO:root:loaded pretrained encoder from epoch: 300 path: weights/paws_imgnt_1percent_300ep.pth.tar
INFO:root:making imagenet data transforms
INFO:root:keep file: imagenet_subsets/99percent.txt
INFO:root:data-path /nfs/ada/hpirsiav/datasets/imagenet/train/
INFO:root:Initialized ImageNet
INFO:root:ImageNet fine-tune dataset created
INFO:root:Using imagenet_subsets/99percent.txt
Namespace(dataset_name='imagenet_fine_tune', device='cuda:0', image_folder='imagenet', model_name='resnet50', normalize=True, pretrained='weights/paws_imgnt_1percent_300ep.pth.tar', root_path='/nfs/ada/hpirsiav/datasets', split_seed=152, subset_path='imagenet_subsets/', unlabeled_frac=0.99, use_pred=True)
weights/paws_imgnt_1percent_300ep.pth.tar
INFO:root:[0/801]
INFO:root:[50/801]
INFO:root:[100/801]
INFO:root:[150/801]
INFO:root:[200/801]
INFO:root:[250/801]
INFO:root:[300/801]
INFO:root:[350/801]
INFO:root:[400/801]
INFO:root:[450/801]
INFO:root:[500/801]
INFO:root:[550/801]
INFO:root:[600/801]
INFO:root:[650/801]
INFO:root:[700/801]
INFO:root:[750/801]
INFO:root:[800/801]
INFO:root:torch.Size([12811, 2048])
INFO:root:torch.Size([12811])
INFO:root:making imagenet data transforms
INFO:root:keep file: imagenet_subsets/0percent.txt
INFO:root:data-path /nfs/ada/hpirsiav/datasets/imagenet/val/
INFO:root:Initialized ImageNet
INFO:root:ImageNet fine-tune dataset created
INFO:root:flip
INFO:root:[ 1200/3125] 64.207% 82.176%
INFO:root:[ 1250/3125] 64.194% 82.124%
INFO:root:[ 1300/3125] 64.109% 82.052%
INFO:root:[ 1350/3125] 64.175% 82.078%
INFO:root:[ 1400/3125] 64.115% 82.026%
INFO:root:[ 1450/3125] 64.085% 82.021%
INFO:root:[ 1500/3125] 64.028% 82.037%
INFO:root:[ 1550/3125] 64.148% 82.068%
INFO:root:[ 1600/3125] 64.030% 82.062%
INFO:root:[ 1650/3125] 64.067% 82.102%
INFO:root:[ 1700/3125] 64.084% 82.110%
INFO:root:[ 1750/3125] 63.996% 82.050%
INFO:root:[ 1800/3125] 64.027% 82.069%
INFO:root:[ 1850/3125] 63.979% 82.060%
INFO:root:[ 1900/3125] 63.966% 82.032%
INFO:root:[ 1950/3125] 63.922% 82.019%
INFO:root:[ 2000/3125] 63.996% 82.031%
INFO:root:[ 2050/3125] 63.923% 82.039%
INFO:root:[ 2100/3125] 63.922% 82.023%
INFO:root:[ 2150/3125] 63.950% 82.046%
INFO:root:[ 2200/3125] 63.994% 82.076%
INFO:root:[ 2250/3125] 64.066% 82.119%
INFO:root:[ 2300/3125] 64.108% 82.144%
INFO:root:[ 2350/3125] 64.114% 82.167%
INFO:root:[ 2400/3125] 64.122% 82.171%
INFO:root:[ 2450/3125] 64.099% 82.168%
INFO:root:[ 2500/3125] 64.132% 82.165%
INFO:root:[ 2550/3125] 64.122% 82.137%
INFO:root:[ 2600/3125] 64.132% 82.122%
INFO:root:[ 2650/3125] 64.146% 82.144%
INFO:root:[ 2700/3125] 64.192% 82.169%
INFO:root:[ 2750/3125] 64.181% 82.218%
INFO:root:[ 2800/3125] 64.174% 82.205%
INFO:root:[ 2850/3125] 64.170% 82.190%
INFO:root:[ 2900/3125] 64.193% 82.211%
INFO:root:[ 2950/3125] 64.131% 82.212%
INFO:root:[ 3000/3125] 64.097% 82.183%
INFO:root:[ 3050/3125] 64.114% 82.180%
INFO:root:[ 3100/3125] 64.133% 82.165%
INFO:root:top1/top5: 64.15/82.164

Thank You Ajinkya Tejankar

MidoAssran commented 3 years ago

Hi @chigur,

I'm looking into this now, and will get back to you shortly! Have you also experienced this issue when running the 10% models?

ajtejankar commented 3 years ago

Hi @MidoAssran,

I did not check the 10% models. Let me try them.

Thank You Ajinkya Tejankar

ajtejankar commented 3 years ago

Hi @MidoAssran,

I was able to reproduce the 10% NN accuracies for all the checkpoints.

Thank You Ajinkya Tejankar

MidoAssran commented 3 years ago

Thanks for letting me know @chigur

So I discovered that the issue was that the 1% NN results were reported using

python snn_eval.py
  --model-name resnet50 --use-pred
  --pretrained $path_to_pretrained_model
  --unlabeled_frac 0.9
  --root-path $path_to_root_datasets_directory
  --image-folder $image_directory_inside_root_path
  --dataset-name $one_of:[imagenet_fine_tune, cifar10_fine_tune]

instead of

python snn_eval.py
  --model-name resnet50 --use-pred
  --pretrained $path_to_pretrained_model
  --unlabeled_frac 0.99
  --root-path $path_to_root_datasets_directory
  --image-folder $image_directory_inside_root_path
  --dataset-name $one_of:[imagenet_fine_tune, cifar10_fine_tune]

The first one uses 10% of imagenet train embeddings for the nearest neighbours evaluation, whereas the second one uses 1% of imagenet train embeddings for nearest neighbours evaluation. The correct one is the second one (similar to the one you used).

I checked the 10% NN results and those seem fine for me as well, and also the 1% and 10% fine-tuned numbers, and those also seem fine, so the issue was localized to the size of the lookup table for evaluating the 1% NN models. Thanks again for noticing this, I will update the table with the correct numbers for the 1% NN models.

MidoAssran commented 3 years ago

I'll close this issue now, but let me know if you have any other questions.

ajtejankar commented 3 years ago

Great! I guess the numbers in the paper are slightly off as well.

MidoAssran commented 3 years ago

yes, the 1% NN need to be changed :) that's actually what I mean by updating the table

facebookresearch / suncet

Unable to reproduce the 1% and 10% NN results #10