search

HobbitLong / RepDistiller

[ICLR 2020] Contrastive Representation Distillation (CRD), and benchmark of recent knowledge distillation methods
BSD 2-Clause "Simplified" License
2.11k stars 389 forks source link

Does the crd can be applied to cross domain distillation #19

Closed Doraemonzm closed 4 years ago

Doraemonzm commented 4 years ago

hi, i wonder if you have applied the crd or cmc to cross domain distillation, for example : vqa or different domains in visual , will it work ?

HobbitLong commented 4 years ago

Hi, @wwwzzzmmm. I tried cross modal distillation. CRD works at least in my scenario. I think CRD and CMC will generally work if you properly construct variables X and Y, and define their joint distribution and product of marginals.

Doraemonzm commented 4 years ago

nice work! thank you