Is it possible to specify different lr for different layer or fix some layer？

[funan-jhc-lee]

Hello！As stated in the title, is it possible to specify different lr for different layer or fix some layer, when retraining a model？I‘ve read the document, but It seems not possible. Thanks a lot for taking the time to answer this question!

[mjwen]

• It's possible to fix the params of some layer (this requires some familiarity with PyTorch). Basically, you can loop over model.named_parameters() and set requires_grad=False for params you want to fix:

  
  model = NeuralNetwork(...) 

do the fitting

...

save the model

load the model back

model.load(model_saved_path)

for name, p in model.named_parameters(): if keyword in name: # keyword is some keyword in the layer you want to fix, e.g. layer1 p.requires_grad = False

- different lr requires multiple optimizers; unfortunately it is not supported out of the box now.