document available lr_schedulers

axolotl-ai-cloud / axolotl

Go ahead and axolotl questions

https://axolotl-ai-cloud.github.io/axolotl/

Apache License 2.0

7.87k stars 866 forks source link

document available lr_schedulers #334

Open winglian opened 1 year ago

winglian commented 1 year ago

class SchedulerType(ExplicitEnum):
    LINEAR = "linear"
    COSINE = "cosine"
    COSINE_WITH_RESTARTS = "cosine_with_restarts"
    POLYNOMIAL = "polynomial"
    CONSTANT = "constant"
    CONSTANT_WITH_WARMUP = "constant_with_warmup"
    INVERSE_SQRT = "inverse_sqrt"
    REDUCE_ON_PLATEAU = "reduce_lr_on_plateau"

tmm1 commented 1 year ago

Is there a bug here, it seems to force cosine:

https://github.com/OpenAccess-AI-Collective/axolotl/blob/c500d025174cacb3dff470e4f1086f84e8f5fb95/src/axolotl/utils/trainer.py#L556-L558

EDIT: never mind, it says NOT in so I just misunderstood