Implement 2 model specific methods

[a-froghyar]

• custom control over the loss dict and the optimizer before the main optimizer step
• custom model specific implementation for gradient clipping

[a-froghyar]

After rebasing my TTS PR, I've realised that the optimizers have also been outsourced to this repo, so this PR also implements a StepwiseGradualLR scheduler. This is a scheduler used to hardcode step-lr pairs in the config the following way:

 lr_scheduler="StepwiseGradualLR",
    lr_scheduler_params={
        "gradual_learning_rates": [
            [0, 1e-3],
            [2e4, 5e-4],
            [4e5, 3e-4],
            [6e4, 1e-4],
            [8e4, 5e-5],
        ]
    },

[a-froghyar]

Ready from my side! Thank you @WeberJulian for the help

[WeberJulian]

why do you need to pass loss_dict and optimizer? They are already reachable by the trainer.

I need to have access to them in the model method --> https://github.com/coqui-ai/TTS/blob/1afd685d230a70b9651d67392b5096dd76042ceb/TTS/tts/models/tacotron2.py#L303

He means that since the first argument of the callback is trainer, in this function, you can call trainer.optimizer, no need to pass it as argument.

Look at how callbacks are implemented in TTS

[WeberJulian]

I'd call this "before_backward_pass"

and move self.model.apply_gradient_clipping in before_gradient_clipping

@erogol do you mean wrapping self.model.apply_gradient_clipping into a separate callback?

No he means move the content of self.model.apply_gradient_clipping inside a new callback called before_gradient_clipping and implement this callback in your model instead of the apply_gradient_clipping function.

[a-froghyar]

Changes implemented in both repos!

[erogol]

I've merged this after rebasing. Thanks for the PR @a-froghyar