search

NVlabs / DIODE

Official PyTorch implementation of Data-free Knowledge Distillation for Object Detection, WACV 2021.
https://openaccess.thecvf.com/content/WACV2021/html/Chawla_Data-Free_Knowledge_Distillation_for_Object_Detection_WACV_2021_paper.html
Other
61 stars 6 forks source link

Image variance collapses to zero #15

Closed karlHolmquist closed 2 years ago

karlHolmquist commented 2 years ago

Hi, thanks for the code, I just want to make sure whether what I'm currently encountering is due to parameter settings or the architecture I'm using.

I've made some modifications to the code so this will not be representative of the original training code but when I'm generating images using a pretrained resnest-101 the generated images quickly reaches a zero-variance (less than 0.01 after 100 iterations). Is this something that often happen or is this a problem that occur due to the differing architecture?

Some potential causes: Different architecture (Resnest-101 + DeepLabv3) All BN layers are used in the loss (I've tried to use the cosine annealing of the number of layers to include) Loss balances (I've tried to completely remove the task loss and only use the task loss + include an image statistics loss matching the mean and variance to a N(0,1) distribution (after normalization))

Thanks beforehand for any insights

karlHolmquist commented 2 years ago

Figured it out, I had missed that I accidentally used weight decay which caused the variance to tend to zero.. Closing the issue