Results Reproducibility

[HaniItani]

Hello,

Thank you for sharing your work.

I'm trying to reproduce ImageNet-C results. I downloaded ImageNet-C from here, and created a dataset using torchvision.datasets.ImageFolder and modified the __getitem__ to allow for is_carry_index.

First, evaluating pretrained ResNet50 (torchvision weights ImageNet1K-V1) on ImageNet-C (level 5) reports 83.11% error rate which is higher than the 82.22% reported in this repo. The error rate on the snow corruption using run_ttac_no_without_queue.sh is around 50% instead of 46.64%. The seed is fixed everywhere in the code so I'm expecting to reproduce both numbers. Any insights as to what might be wrong? I'm using PyTorch 1.12.1 and CUDA 10.2.89.

I reproduced the results on CIFAR-10 successfully.

Your suggestions and insights would be really appreciated.

Best regards, Hani

[Dyb3438]

Hi Hani,

Thank you for your interest in this work.

The ImageNet-C used in our code is generated by create_corruption_dataset.py. In this script, we leverage the package from here to generate various corrupted sets and save them as "*.pth". You can refer to Readme.md for this specific operation process.

Also, I again ran the approximate results as the values reported in this repo, as shown below. 82.22% is the directly testing error rate; 46.54% is the TTAC (without queue) error rate. The partial output results are as follows.

BATCH: 351/391 instance error: 0.46966257122507127
BATCH: 352/391 instance error: 0.46939364346590906
BATCH: 353/391 instance error: 0.4692590297450425
BATCH: 354/391 instance error: 0.4691031073446328
BATCH: 355/391 instance error: 0.4692781690140845
BATCH: 356/391 instance error: 0.4692108497191011
BATCH: 357/391 instance error: 0.469187675070028
BATCH: 358/391 instance error: 0.4693173882681564
BATCH: 359/391 instance error: 0.46916347493036215
BATCH: 360/391 instance error: 0.46898871527777775
BATCH: 361/391 instance error: 0.46892313019390586
BATCH: 362/391 instance error: 0.4688794889502762
BATCH: 363/391 instance error: 0.4685993457300276
BATCH: 364/391 instance error: 0.4686212225274725
BATCH: 365/391 instance error: 0.4683219178082192
BATCH: 366/391 instance error: 0.46823770491803274
BATCH: 367/391 instance error: 0.4683242506811989
BATCH: 368/391 instance error: 0.4684952445652174
BATCH: 369/391 instance error: 0.46830538617886175
BATCH: 370/391 instance error: 0.4682221283783784
BATCH: 371/391 instance error: 0.4680129716981132
BATCH: 372/391 instance error: 0.46797295026881724
BATCH: 373/391 instance error: 0.4677446380697051
BATCH: 374/391 instance error: 0.4674757687165776
BATCH: 375/391 instance error: 0.4672708333333333
BATCH: 376/391 instance error: 0.46729554521276595
BATCH: 377/391 instance error: 0.46711289787798405
BATCH: 378/391 instance error: 0.46713789682539686
BATCH: 379/391 instance error: 0.4669978562005277
BATCH: 380/391 instance error: 0.4667557565789474
BATCH: 381/391 instance error: 0.46665846456692917
BATCH: 382/391 instance error: 0.4665003272251309
BATCH: 383/391 instance error: 0.4666489882506527
BATCH: 384/391 instance error: 0.46638997395833337
BATCH: 385/391 instance error: 0.4660917207792208
BATCH: 386/391 instance error: 0.4661188471502591
BATCH: 387/391 instance error: 0.4658833979328165
BATCH: 388/391 instance error: 0.4656692976804123
BATCH: 389/391 instance error: 0.4655567159383034
BATCH: 390/391 instance error: 0.46544471153846156
BATCH: 391/391 instance error: 0.46536
snow Test time training result: 0.46536

I'm using CUDA 11.1, PyTorch 1.10.0, Python 3.6.2 and RTX 3090.

Finally, I hope this reply is helpful to you and I kindly suggest you could download and run this released code firstly.

Yongyi

[HaniItani]

Hello,

Thank you for your reply.

I generated the snow corruption from ImageNet and I was able to reproduce your numbers. I got 82.10% error rate using ResNet50 and 46.172% error rate after adaptation.

Thank you for your suggestion. Best reagards, Hani