IAHispano / Applio

A simple, high-quality voice conversion tool focused on ease of use and performance
https://applio.org
MIT License
1.81k stars 292 forks source link

optimizer step bugfix #860

Closed codename0og closed 2 weeks ago

codename0og commented 2 weeks ago

Fixed the incorrect use of scalers in relation to optim steps

Description

Simply remove both:

optim_g.step()
optim_d.step()

at lines: 433 and 434

Motivation and Context

RVC and Applio use AMP and so, GradScaler and autocast. Hence: You should not use optim.step() directly when you're working with GradScaler. If you do, you would bypass the necessary gradient scaling and unscaling steps, which can lead to suboptimal training performance and potential instability. In other words: It is just simply incorrect and one should use either scaler.step(optim) if AMP/autocast/scaler is in use or optim.step() if there's no plans for mixed precision training.

How has this been tested?

Debugging NaNs, Infs and values/params in gradients of Generator and Discriminator.

Types of changes

[bugfix]

Checklist:

AznamirWoW commented 2 weeks ago

added the same change to my PR