search

microsoft / mup

maximal update parametrization (µP)
https://arxiv.org/abs/2203.03466
MIT License
1.24k stars 88 forks source link

Interpreting jitter in coordcheck #58

Closed leenachennuru closed 6 months ago

leenachennuru commented 9 months ago

Hi,

I have coord check plots with mup that look roughly stable but are a bit more jittery compared to the example plots. I was wondering if this can be considered expected behavior.

mup_test

Thanks!

edwardjhu commented 8 months ago

It's good as long as the magnitude is stable. One way to verify further is to extend the x-axis to the right

leenachennuru commented 6 months ago

Thanks! I could verify the stability of LR with width even with this jitter. Closing this issue accordingly.