Learning rate schedulers

[dmccloskey]

Description

Methods to allow for increasing or decreasing the learning rate based on user defined criteria during training

Objectives

• [ ] Time-based decay method for ModelTrainer
• [ ] Step decay method for ModelTrainer
• [ ] Exponential decay method for ModelTrainer
• [x] Plateau decay method for ModelTrainer
• [ ] Unit tests for for ModelTrainer

Validation

• [ ] Passing unit tests

Rerences

Time-based decay

lr = lr0/(1+kt)
lr *= (1. / (1. + self.decay * self.iterations))

Step decay

lr = lr0 * drop^floor(epoch / epochs_drop)
def step_decay(epoch):
   initial_lrate = 0.1
   drop = 0.5
   epochs_drop = 10.0
   lrate = initial_lrate * math.pow(drop,  
           math.floor((1+epoch)/epochs_drop))
   return lrate

Exponential decay

lr = lr0 * e^(−kt)
def exp_decay(epoch):
   initial_lrate = 0.1
   k = 0.1
   lrate = initial_lrate * exp(-k*t)
   return lrate