search

kiwi12138 / RealisticTTA

Official repository for AAAI2024 paper <Unraveling Batch Normalization for Realistic Test-Time Adaptation>.
6 stars 1 forks source link

Question about Table 2. #2

Open waychin-weiqin opened 4 months ago

waychin-weiqin commented 4 months ago

Hi,

Thank you for sharing the interesting work! I have a question regarding the results presented in Table 2.

I am wondering if these results were generated using a single corruption sequence. I have tried re-running Tent using the sequence found in your config file and ended up with very different results. For example, in Table 2, Tent trained with a batch size of 64 shows an error rate of 62.60, but I have obtained an error rate of 91.0.

Do you have any insights on this matter?

Thank you and I look forward to your reply!

kiwi12138 commented 4 months ago

Dear weiqin,

Did you follow the original code from https://github.com/mariodoebler/test-time-adaptation/tree/main? Some of our modules are changed to cope with our designed strategy so if you would like to produce the results for the compared method, you are suggested to directly use the original code.

Also, you may follow the benchmarks from their page to compare with your results: https://docs.google.com/spreadsheets/d/1xR-3df5xMMsEcMHe4Vo495E35RrIPs5abcR7Pztvucw/edit?usp=drive_link. image

waychin-weiqin commented 4 months ago

Thank you for your swift reply!

I managed to get similar results using the original code from the link you shared. Also, thank you for sharing the spreadsheet.

By any chance do you know the configs (learning rate and batch size) for running ViT or Swin for ImageNet-C using TENT?

Thank you again for your help :)

kiwi12138 commented 4 months ago

You're very welcome! I'm glad to hear that you managed to achieve similar results using the original code.

Regarding your query about the configurations for running ViT or Swin on ImageNet-C using TENT, I haven't personally experimented with the ViT or Swin backbones in this context. However, you might find useful insights in the paper, ROID, from the original code's page, which discusses some related configs. Additionally, I would recommend directly reaching out to the authors by commenting in the code repository, as they might be able to provide specific details and tips.

Thank you for reaching out, and I wish you the best of luck with your project!