LRRT w/ Validation

[phelps-matthew]

Hi there, very much enjoyed the blog article. I was wondering if you may be able to share the Keras implementations of the LR finder w/ the validation curve as well. Understandably this will take much longer to compute, but based on https://arxiv.org/pdf/1803.09820v2.pdf, the validation loss provides some valuable clues in identifying behavior and for setting the bounds in CLR.

Thank you!

[simonamaggio]

Hi @phelps-matthew, thank you for your interest. I will include this option later in this code, after some cleaning, but in the meanwhile what you need to modify to have the validation curve is:

• In the LRFinder callback here modify:

loss = logs.get('val_loss')
acc = logs.get('val_accuracy')

• In the training (notebook) make sure to run the evaluation at each batch:

model.fit_generator(data_gen,
                    steps_per_epoch=NUM_ITERATIONS_PER_EPOCH,
                    epochs=NUM_EPOCHS,
                    callbacks=[lr_finder],
                    shuffle=rnd_batch,
                    validation_steps=NUM_ITERATIONS_PER_EPOCH) # **HERE**

[phelps-matthew]

Thank you kindly! I was able to implement by adding in a model.evaluate portion within on_train_batch_end. As expected, the validation steps is a multiplier on how much longer the learning rate finding takes, but I have found it to be helpful for selecting the best lr range.