Training After Switch to SGD is Flat

[noiran78]

Hi,

Thanks for the implementation of SWATS in this repository. However, when I used it on VGG-16 running for 100 epochs with initial learning rate 0.001, after switching to SGD on epoch 15, the learning curve fell significantly and it stays flat. The updated learning rate is 0.34.

Is this an expected behavior or is there any problem here?

Looking to hear from you.

Best Regards

[Mrpatekful]

Sorry for the late response, I haven't received notification of this issue. SGD generally requires some kind of lr scheduling, which is not included in this implementation. I tried adding it, however I couldnt find a nice solution to handle the SGD and Adam phase differently, which is desirable for any kind of scheduling.

[jonathan016]

Oh I see. So in my case, what do you think should be done?

I'm facing similar issue (flat accuracy after switching)

[Mrpatekful]

You could try writing some kind of custom lr scheduling, since the optimizer stores the current phase in the param group with the key phase and apply different behavior based on that.

def custom_schedule(optimizer, compute_adam_lr, compute_sgd_lr):
    for param_group in optimizer.param_groups:
        if param_group['phase'] == 'SGD':
            param_group['lr'] = compute_sgd_lr()
        elif param_group['phase'] == 'ADAM':
            param_group['lr'] = compute_adam_lr()

[jonathan016]

I see. Thanks a lot, will try that and hopefully the problem does not occur again. Thank you @Mrpatekful