search

yaoxufeng / PCL-Proxy-based-Contrastive-Learning-for-Domain-Generalization

MIT License
57 stars 6 forks source link

Results w/o SWAD #4

Open korts-lew opened 2 years ago

korts-lew commented 2 years ago

Hello, I want to say thank you for your great work first! I make this issue for asking a question related to https://github.com/yaoxufeng/PCL-Proxy-based-Contrastive-Learning-for-Domain-Generalization/issues/1

In these days, I'm trying to compare DG algorithms based on DomainBed.

So, I want to know performances of PCL w/o SWAD for datasets you've tested (PACS, VLCS, OfficeHome, TerraIncognita, DomainNet) to compare w/ algorithms not using SWAD

Can you show the results for them? (i.i.d. results of log.txt)

It would be better if they are results of several different runs

If it is impossible or hyperparameters can be different from one w/o SWAD, can you share hyperparameters you've used for them? (as far as i know, i can not check hyper-parameters for VLCS and DomainNet in this repo)

Thanks

yaoxufeng commented 2 years ago

Hi Byounggyu-Lew,

Thanks again for your interest in our work. It's a good question! Our work is largely based on SWAD and we also observed the hyperparameters of PCL are sensitive. We believe there exists a large room for improving it. We tend to believe PCL can also work based on other codebase, however, it may require some time to find other hyperparameters.

Overall, thanks again for raising this question, which also inspires us to explore more in DG.

korts-lew commented 2 years ago

Thank you for your kind reply.

First of all, I wasn't talking about the codebase. I can do the test on SWAD codebase too. What I want to know is the performance of PCL without applying SWAD in model selection. As table in original SWAD paper shows, CORAL + SWAD is much better than vanilla CORAL. However, tables in your paper just compare PCL against CORAL, not CORAL + SWAD. It seems a little bit unfair for me. But, I don't have a doubt to your method and I appreciate the novelty of your paper. I just want to compare the performances more fair setting for me.

BTW, as far as I've understood, I think you're saying that training PCL w/o SWAD would require different hyper-parameters compared to the one w/ SWAD for optimal performance and finding such hyper-parameters might be somewhat hard due to sensitivity. In other words, comparing PCL w/o SWAD with other methods based on current hyper-parameters used in the paper would be unfair too. Am I right?

Thank you again for your great work 😄

BierOne commented 1 year ago

Hi, @Byounggyu-Lew @yaoxufeng ! Thanks for this good question. I am also interested in the results w/o SWAD on the Domainbed benchmark. Are there any updates?