LRFinder() modified to start from the initial learning rate user provided.

[yongduek]

I was looking for a pytorch lrfinder and came to see your tutorials. Very nice and thank you for sharing them.

When lrs from LRFinder was printed out (AlexNet.ipynb), it did not start with 1e-7. So two things seem to be modified to make it happen.

1. Order change: get_lr() before append to lrs

2. Function change: 'get_last_lr()` to get the latest lr. This seems to be a change in pytorch.

           lrs.append(lr_scheduler.get_last_lr()[0])
   
           #update lr
           lr_scheduler.step()

3. self.last_epoch does not need to be incremented.

   class ExponentialLR(_LRScheduler):
   def get_lr(self):
       curr_iter = self.last_epoch # + 1
       r = curr_iter / self.num_iter
       return [base_lr * (self.end_lr / base_lr) ** r for base_lr in self.base_lrs]

   • From inspection, super().__init__() calls step() within _LRScheduler, then step() calls get_lr() to update the values. The results are saved in variables in the base class (_LRScheduler), and the lastly updated learning rate (or list of lrs) are retrieved by get_last_lr(). This seems to be the recent pytorch way of using it.

[bentrevett]

Great catch and thanks for spotting!

The LR finder code was a modified version of https://github.com/davidtvs/pytorch-lr-finder/. It looks like they also use get_lr and add the lr to their history after the step too, see here.

Might be worth raising an issue on there as well?