Closed nousr closed 2 years ago
it is normal i believe, you can always try lowering the ema_beta
on the Trainer to 0.995
or even 0.99
to have it update faster
gotcha, glad I asked then! just have to be patient then 😄
@lucidrains hey! when you get a chance, do you think could you explain the intuition for this line?
specifically, why the // self.update_every
? I guess I don't see the purpose of moving self.update_after
"up" by a factor of update_every
?
Re-opening this issue as @Veldrovive has confirmed that the EMA is not giving expected results for the decoder. A discussion with Katherine has confirmed my suspicions as well.
Will try to do some more debugging after my class final today.
@nousr oh crap, yes, you are right! i thought that i wasn't incrementing the EMA steps in accordance with the global training step, but i was
https://github.com/lucidrains/DALLE2-pytorch/commit/9cc475f6e7990b3d978128902ee0ea90614451f6 should fix the update every issue
So, I've noticed a potential bug related to the exponential moving averaged priors (I haven't tested the other models).
Essentially, the EMA model seemingly refuses to "learn". I have to believe that the weights aren't being properly updated with the
online_model
, for whatever reason.As you can see, as soon as the EMA kicks in at step 1000, the loss plateaus. I've never worked with an EMA model before, so I guess this could be expected behavior?
This issue also presents itself when running evaluation on the EMA model (the
(text<->predicted_image)
similarity should be at least 0.1 even after a few hundred steps)