Inquiry Regarding Training Procedure and Model Selection on Learning Rate Adjustment

[chuanlukk]

Question: When the learning rate is adjusted after early stopping, why does the code continue training from the latest saved model instead of reloading the best-performing model? Is this behavior intentional, or could it be a bug?

    early_stopping(acc, model)
    if early_stopping.early_stop:
        cont_train = model.adjust_learning_rate()
        if cont_train:
            print("Learning rate dropped by 10, continue training...")
            early_stopping = EarlyStopping(patience=opt.earlystop_epoch, delta=-0.002, verbose=True)
        else:
            print("Early stopping.")
            break
    model.train()

[chuanlukk]

It is possible to continue training an overfitted model.

[PeterWang512]

In our experience, we didn't observe overfitting with this updating strategy. However, you can feel free to change this code to test if it improves performance. Thanks!