Why use 3 optimizers during training?

[Ther-nullptr]

Hi! I want to ask why we should use 3 optimizers during training? I think self.optimizer.zero_grad() is enough.

self.optimizer.zero_grad()
if self.l0_optimizer is not None:
     self.l0_optimizer.zero_grad()
if self.lagrangian_optimizer is not None:
     self.lagrangian_optimizer.zero_grad()

[xiamengzhou]

These three optimizers are used to update the parameters of different model parts. self.optimizer is used to update the main model parameters; self.l0_optimizer is used to update the parameters of the L0 module and self.lagrangian_optimizer is used to update the lagrangian parameters ($\lambda_1$ and $\lambda_2$). The learning rates used for these optimizers are different.

[Ther-nullptr]

get it!