search

facebookresearch / DomainBed

DomainBed is a suite to test domain generalization algorithms
MIT License
1.42k stars 298 forks source link

Would it be possible to run a new sweep? #82

Closed cwognum closed 2 years ago

cwognum commented 3 years ago

The last sweep is from 14 months ago and since then 10 new algorithms have been added. These newly added algorithms generally present only partial or no results at all on DomainBed. To the best of my knowledge, Fishr (Table 4) includes the most up-to-date sweep, but it still lacks 6 of the currently available algorithms.

I would be curious to get an updated overview of how the current set of algorithms compare to one another, but I personally do not have access to the compute to run such a sweep. Would it by any chance be possible for you to schedule a new sweep sometime soon?

lopezpaz commented 2 years ago

We are indeed planning it, but given the amount of compute, we are first considering some refactoring of the code. We will update the repository as soon as we have new results available.