Fixes to schedules and DDP - Githubissues

LoicGrobol / zeldarose

Train transformer-based models.

https://zeldarose.readthedocs.io

Other

28 stars 3 forks source link

Fixes to schedules and DDP #9

Closed LoicGrobol closed 3 years ago

LoicGrobol commented 3 years ago

Changed

max_steps is automatically inferred from the tuning config if a number of lr decay steps is given
max_epochs is not optional (if both max_steps and max_epochs are unset and no lr schedule is provided, Lightning's default will be used)
find_unused_parameters is now disabled in DDP mode, unless in profile mode

Fixed

Linear decay now properly takes the warmup period into account