search

zhengzangw / DoPrompt

Official implementation of PCS in essay "Prompt Vision Transformer for Domain Generalization"
MIT License
51 stars 4 forks source link

About hyperparameters #1

Open dltkddn0525 opened 1 year ago

dltkddn0525 commented 1 year ago

Hi, thanks for sharing your great works.

I'm trying to reproduce the result of ERM(3rd row in Table.1 from your paper), but I got a bit lower performance on OfficeHome dataset. I ran your source code with hyperparameters described in your paper(lr=1e-05, weight_decay=1e-02, last dropout=0.1), and I got 72.767% of target accuracy averaged on 3 different random seeds. I think I'm missing some other hyperparameters such as classifier LR multiplier or attention dropout probability in vision transformer which are not clarified in the paper.

Could you kindly share the searched hyperparameters on each dataset?

zhengzangw commented 1 year ago

Sorry for the late reply.

We look into the hyper-parameters for ERM, and find the best choice is not the same as our methods. Typically, the classifier LR multiplier should be smaller to get a better result. The command has been updated in the README.md.

dltkddn0525 commented 1 year ago

Thanks. I'll try again with the updated command.

By the way, I found that hparams['lr_classifier'] does not affect the learning rate in the optimizer for ERM in https://github.com/zhengzangw/DoPrompt/blob/main/domainbed/algorithms.py#L111. I think this should be fixed.

zhengzangw commented 1 year ago

Thank you for pointing this out. I have modified the code to match our experiment version.