search

lessw2020 / Ranger21

Ranger deep learning optimizer rewrite to use newest components
Apache License 2.0
320 stars 44 forks source link

Error when using DDP #49

Open smartbarbarian opened 1 year ago

smartbarbarian commented 1 year ago

I used ZeroRedundancyOptimizer to wrap Ranger21 and running it in a 4-GPU machine. But its performace is much worse compared with the simple AdamW and it showed the error. loerror in warmdown pct calc. new pct = 67.11272727272727���������������������������������������������������������������� | 19156/19505 [8:57:59<04:55, 1.18it/auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue! error in warmdown pct calc. new pct = 67.11272727272727 auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue! error in warmdown pct calc. new pct = 67.11272727272727 auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue! error in warmdown pct calc. new pct = 67.11272727272727 auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue!

TolerantChief commented 1 year ago

Did you find a solution to this? I have the same problem.