search

DequanWang / tent

ICLR21 Tent: Fully Test-Time Adaptation by Entropy Minimization
https://arxiv.org/abs/2006.10726
MIT License
344 stars 43 forks source link

Cifar-10c, Cifar-100c and Imagenet-c Results (Not as reported in the paper) #9

Closed jmiemirza closed 1 year ago

jmiemirza commented 3 years ago

Hi,

Thank you for a very nice paper. I have tried to reproduce (or get close) to the results mentioned in the paper. I cannot reproduce the results for Cifar 10c and Cifar 100c by using the resnet-26 as also used by TTT (Sun et. al) paper. I am pretty sure, that I am doing everything right because my trained (resnet-26) gives me an error of 8.2 percent on the clean test set.

When I try to use it with tent, the performance degrades. I am attaching the log file for Resnet 26 here with this issue just so you know what is happening. I must also mention that the results mentioned in the paper (for Cifar10c and 100c) were reproduced by using a Wide resnet 28. But there is NO mention of using a wide resnet 28 in the paper.

Did you update the final version of the paper or I am doing something wrong? I am also attaching the log file with this issue for the results obtained by using wide resnet 28.

Also, can you please tell me what architecture did you use for the imagenet results? Basically when I take an off the shelf resnet 50 (pretrained on imagenet) and try to reproduce the results. It doesn't. In fact the system totally blows up! Is it some other model used for imagenet results as well, instead of resnet 50? Also, adding the log file for Imagenet results.

I look forward to your answer.

Thanks and BR,

Mirza.

tent_resnet_26.txt tent_wide_resnet_28.txt tent_imagenet.txt

thomaspzollo commented 2 years ago

@jmiemirza did you ever get this to work? I'm trying to use this with ResNet50 backbones and performance is very bad (e.g. model that has 8% error on cifar-10 and 22% error on cifar-10-c goes to 70%+ error).

jmiemirza commented 2 years ago

@thomaspzollo Hi, No I did not ever get it to work for ImageNet-C. However, I was able to reproduce the results for Cifar-10C but by using the wide-resnet backbones (from robustbench) mentioned in the repo.

thomaspzollo commented 2 years ago

@jmiemirza thanks for getting back! so you never applied this on a resnet50 backbone?

jmiemirza commented 2 years ago

@thomaspzollo No. Only for the WRN backbones.

wangtingwei1993 commented 2 years ago

@thomaspzollo @jmiemirza Hi, I think ResNet26 and ResNet50 are not officially supported in the RobustBench project (ThreatModel.common_corruption). Do you define the model by yourselves? Did you train the standard ResNet26/50 on cifar10 by yourselves? Thanks in ahead!