search

chaosparrot / parrot.py

Computer interaction using audio and speechrecognition
MIT License
139 stars 36 forks source link

Implement a learning rate schedule #18

Open pokey opened 1 year ago

pokey commented 1 year ago

Currently, it appears learning rate is constant. My loss started to really slow down; I think dropping it could help a lot, but there is no schedule, and due to #15 and #17, I couldn't just restart it with a smaller LR

See #29, which has a few lines that implement a LR schedule that could be extracted for a PR here

ym-han commented 1 year ago

As a related quick note to self: In addition to implementing some sort of learning rate scheduler, we should also add a learning rate finder. Lightning seems to ship with one: https://pytorch-lightning.readthedocs.io/en/1.4.5/advanced/lr_finder.html

pokey commented 1 year ago

yes was thinking a lr range test would be a good idea; forgot that was automated in lighning. good stuff