CelebA experimental setup

[alexrame]

Thanks again for the nice work. I have got few questions regarding the experimental setup for CelebA.

In DomainBed/sweep/CelebA_Blond/run.sh, why set holdout_fraction to 0.1, rather than 0.2 as stated in the arXiv paper ? In DomainBed/sweep/CelebA_Blond/hparams.json, why set resnet_dropout to False ? and why not freeze the batch norms as in DomainBed ? In DomainBed/domainbed/datasets.py#L898, why set the number of training steps to 2000 (rather than 5000) ?

This may seem minor, but clarifying these choices may improve the usability of your benchmark. Best regards, Alexandre

[m-Just]

In DomainBed/sweep/CelebA_Blond/run.sh, why set holdout_fraction to 0.1, rather than 0.2 as stated in the arXiv paper ?

All the experiments (not only those on CelebA) are conducted with holdout_fraction=0.1. This is a typo in our paper. We think 0.2 might be too large for small datasets like PACS and NICO.

In DomainBed/sweep/CelebA_Blond/hparams.json, why set resnet_dropout to False ? and why not freeze the batch norms as in DomainBed ?

We set resnet_dropout to False to further limit the hyperparameter search space (which is quite large for some algorithms). Again, other datasets also follow this setting in our experiments. Empirically, we found this reduces the variance in model performance and increases the performance of the best model found by the search (in many cases). We intentionally didn't follow DomainBed on the freezing of batch norms because most of the algorithms did not freeze the batch norms as they were proposed in the original papers.

In DomainBed/domainbed/datasets.py#L898, why set the number of training steps to 2000 (rather than 5000) ?

To speed up the search process. In our preliminary experiments, we found that the network often converged after only a few hundred steps on this dataset.

Hope these are helpful!