search

facebookresearch / DomainBed

DomainBed is a suite to test domain generalization algorithms
MIT License
1.42k stars 298 forks source link

cdpl parameters in SelfReg are not registered in the optimizer #62

Closed kowshikthopalli closed 3 years ago

kowshikthopalli commented 3 years ago

In SelfReg algorithm, cdpl is defined in L 1031-1040. However these parameters are not appended to parameters in self.network or self.optimizer. This leads to no update in self.cdpl whatsoever. I went through the paper's main code base independent of domainbed at https://github.com/dnap512/SelfReg/blob/334af5fc2430d4183210956bc4d86d86816feeee/codes/model/resnet18_selfreg.py#L174 and observed that self.proj which is self.cdpl is directly registered to the model and hence optimizer will optimize these parameters too. Thank you

dnap512 commented 3 years ago

Hi, @kowshikthopalli !

First of all, thank you for your interest in our research.

It is true that the experiment was conducted with the current DomainBed code. We also experimented with the CDPL initialized (untrained) in our code, and found interesting results. The results are as below. (Data: PACS, Model: ResNet18) image

It is found that the performance difference between (with trained CDPL) and (with untrained CDPL) is not significant. Maybe it's because SelfReg is not big enough to train CDPL. We are currently discussing these results.

Thank you.

lopezpaz commented 3 years ago

@kowshikthopalli Thank you for opening the issue.

@dnap512 Thank you for the clarification. Should we close?

dnap512 commented 3 years ago

@lopezpaz Yes, Thank you.

kowshikthopalli commented 3 years ago

@dnap512 @lopezpaz thank you for your efforts in replying to me in a timely manner. @dnap512 please update us if you can when you finish analyzing the results. A minor comment- why is cdpl called class-dependent ? There are no multiple cdpl s, nor does SelfReg impose any conditions on the class labels to apply a cdpl layer. It is thus just a projection layer right?

dnap512 commented 3 years ago

@kowshikthopalli We will update our analysis to you soon. You know, the CDPL is only trained by SelfReg, not classification loss. And SelfReg is MSEloss between positive pairs (the same class samples). Thus, we designed the CDPL to slightly perturb distribution of features, while preserving class-specific information. But, on current condition, yes, it's a projection layer.

dnap512 commented 3 years ago

@kowshikthopalli

We experimented by increasing the learning rate of the CDPL, and confirmed the highest performance when the learning rate was 15 times larger than the existing learning rate. As mentioned before, it is probably that SelfReg is not large enough to train the CDPL. The results are as below. 이슈용

If you have any questions about our work, please use this repository.

Thank you for your interest!