search

DequanWang / tent

ICLR21 Tent: Fully Test-Time Adaptation by Entropy Minimization
https://arxiv.org/abs/2006.10726
MIT License
344 stars 43 forks source link

The results on ImageNet-C sensitive to some hyperparameters #4

Open ghost opened 3 years ago

ghost commented 3 years ago

Hi, thanks for sharing your great work. And the current repository only contains example code to illustrate how tent works. I am wondering if you will share the code to exactly reproduce results on ImageNet-C or some implementation details if the code is not available. Because it seems that tent is very sensitive to choices of some hyperparameters.

shelhamer commented 3 years ago

Thanks for your interest in tent ⛺ !

I am wondering if you will share the code to exactly reproduce results on ImageNet-C

The code for ImageNet-C will be included shortly, once we have finished simplifying it and re-running it. I'll follow up here and close this issue when it's pushed.

or some implementation details

Please see page 5 of the paper at ICLR'21 for some details of the hyperparameters:

optimization hyperparameters

it seems that tent is very sensitive to choices of some hyperparameters

Could you tell us which specific hyperparameter/s you would like to know about?

For the optimization settings, we have seen improvements with a variety of learning rates [0.00025, 0.01] with SGD+momentum or Adam. However, the amount of improvement can vary, and there are settings that hurt. We recommend selecting hyperparameters on the held-out "extra" corruptions (speckle, spatter, gaussian_blur, saturate).

For the model, we have used the pre-trained ResNet-50 model from pycls as our baseline, as well as ResNet-50 models that we have trained ourselves.

DLwbm123 commented 3 years ago

Hi! Could you please also share the code used for the segmentation experiments? Thanks!

XuanPu-Z commented 1 year ago

Thanks for your interest in tent ⛺ !

I am wondering if you will share the code to exactly reproduce results on ImageNet-C

The code for ImageNet-C will be included shortly, once we have finished simplifying it and re-running it. I'll follow up here and close this issue when it's pushed.

or some implementation details

Please see page 5 of the paper at ICLR'21 for some details of the hyperparameters:

optimization hyperparameters

it seems that tent is very sensitive to choices of some hyperparameters

Could you tell us which specific hyperparameter/s you would like to know about?

For the optimization settings, we have seen improvements with a variety of learning rates [0.00025, 0.01] with SGD+momentum or Adam. However, the amount of improvement can vary, and there are settings that hurt. We recommend selecting hyperparameters on the held-out "extra" corruptions (speckle, spatter, gaussian_blur, saturate).

For the model, we have used the pre-trained ResNet-50 model from pycls as our baseline, as well as ResNet-50 models that we have trained ourselves.

Hi! I'm a new student on test-time adaption and very fond of your work. Are you ready to share the code for ImageNet-C yet? Thanks!