search

openai / guided-diffusion

MIT License
6.06k stars 807 forks source link

Bug in EMA calculation? #26

Closed coleridge72 closed 2 years ago

coleridge72 commented 2 years ago

Currently the EMA params are calculated as:

https://github.com/openai/guided-diffusion/blob/27c20a8fab9cb472df5d6bdd6c8d11c8f430b924/guided_diffusion/nn.py#L65

I think it should actually be swapped to:

targ.detach().mul_(1 - rate).add_(src, alpha=rate)

I looks like this is currently just taking an average of < 1 step when rate=0.9999

Source: https://fastai.github.io/timmdocs/training_modelEMA

coleridge72 commented 2 years ago

Ah, actually ignore me... this is not a bug, I was just mixed up between what was source params and what was target.