search

ryoungj / optdom

[ICLR'22] Self-supervised learning optimally robust representations for domain shift.
MIT License
23 stars 3 forks source link

Why there is a performance gap between the CAD in this repo and the Domainbed repo #2

Closed BierOne closed 2 years ago

BierOne commented 2 years ago

Hi @ryoungj! Thanks for this outstanding work. I really appreciate your public code.

Recently, I am trying to reproduce the results of CAD (based on ResNet-50 backbone) in your paper. But I found a performance gap between this repo (83.5) and the domainbed repo (79.5) in PACS. In detail, for both repositories, I swept 40 hyper-params already. And in this repo, I use the run_sweep_e2e_domainbed.sh.

My questions are

Pls correct me if there is something wrong. Thank you so much.

ryoungj commented 2 years ago

Hi,

Thanks for your interests in our paper & code! I would try my best to address your questions:

  1. I'm not totally sure about the exact reason because I'm not following recent updates in the DomainBed repo anymore. But one potential reason might be the hyper-parameter selection used. In particular, if you use the 'source selection' method, I would expect the results to have very high variance (across differently sampled hyper parameters), and sampling more hyper-parameters could lead to worse target performance (because the selected model could overfit to the source). As for which code to use, I would recommend using ours if you would like to study CAD because I haven't tracked latest updates there that could possibly make some differences. Another potential reason is that it seems that you might use CAD instead of CondCAD (see below).
  2. In our paper, we mainly introduce CAD in the main body for simplicity (because it is not the main focus), while mentioning that "For the CAD bottleneck we used its conditional version whenever labels were available". So in DomainBed where labels are provided, we used CondCAD. I apologize if it leads to some confusions.
  3. Yes, they are based on ResNet-50 (but CondCAD).

Let me know if you have any further questions!

BierOne commented 2 years ago

Thank you so much for this prompt reply. It solves my problems nicely! For your information, I have tried the CondCAD (ResNet50 backbone) with this repo, which achieves your reported performance (85.5) on PACS.

BierOne commented 2 years ago

Hi, @ryoungj! It seems that the pretrained CLIP model is not really finetuned in practical. Is this right? -- All of the finetuning is performed on the MLP layer in here.

Again, thanks for your time.

ryoungj commented 2 years ago

That is correct. In the experiment setup in out paper (Sec 6.2), we described that

We did so in two simple steps. First, we froze the pretrained CLIP and added a multi-layer perceptron (MLP) that could effectively finetune CLIP’s representations. Then, we trained the MLP by minimizing our CAD bottleneck...