search

facebookresearch / dadaptation

D-Adaptation for SGD, Adam and AdaGrad
MIT License
501 stars 19 forks source link

Complex number not supported for max() operation #15

Closed jinyi12 closed 1 year ago

jinyi12 commented 1 year ago

Hi, thanks for adding complex support. However for my own dataset, I am getting the same error at d = max(d, min(d_hat, d*growth_rate)) in dadapt_adam.py. Upon inspecting using the debugger, I found that d_hat is complex valued, hence the error.

adefazio commented 1 year ago

We might be using different PyTorch versions, as the complex number support has changed over time. What version are you on? When I run the code you linked me to all the values passed into that operation are real valued. Can you tell which of sksq_weighted, gsq_weighted and sk_l1 are complex?

jinyi12 commented 1 year ago

Hi @adefazio, thanks for the prompt reply, and I really enjoyed using dadaptation in my other project.

I am using Pytorch 2.0 for the moment.

Both sksq_weighted and gsq_weighted are complex in my case. Before gsq_weighted becomes complex, g_sq is complex, hence gsq_weighted = beta2gsq_weighted + g_sq(dlr*2)(1-beta2) results in a complex gsq_weighted.

adefazio commented 1 year ago

Unfortunately I think I'm going to have to mark this as "won't fix". I'm not able to debug the issue and it seems like a low-priority use case. Sorry!