search

ruizhecao96 / CMGAN

Conformer-based Metric GAN for speech enhancement
MIT License
309 stars 60 forks source link

the change of gen_loss during training #29

Closed BancoLin closed 1 year ago

BancoLin commented 1 year ago

Due to limited GPU memory I have to change following parameters to reduce memory usage in training:

Training works fine and the gen_loss in the checkpoint model filename seems to be stuck about 0.057. Just curious about what the gen_loss would be if I use the default parameters.

SherifAbdulatif commented 1 year ago

Here is a part of the logs and I think it fluctuates: "INFO:root:Epoch 0, Step 2000, loss: 0.0698222815990448, disc_loss: 0.003855810035020113 INFO:root:Epoch 0, Step 4000, loss: 0.10153232514858246, disc_loss: 0.0027087130583822727 INFO:root:Epoch 0, Step 6000, loss: 0.09570753574371338, disc_loss: 0.013898389413952827 INFO:root:Epoch 0, Step 8000, loss: 0.1152464747428894, disc_loss: 0.003877736860886216 INFO:root:Epoch time: 5668.203575849533 INFO:root: pesq: 2.9439869927276034, csig: 4.347757358511038, cbak: 3.57811086328908, covl: 3.7087207256078827, ssnr: 9.00936124113099, stoi: 0.94614695583022

INFO:root:Epoch 1, Step 2000, loss: 0.07784414291381836, disc_loss: 0.020618950948119164 INFO:root:Epoch 1, Step 4000, loss: 0.061434850096702576, disc_loss: 0.043131016194820404 INFO:root:Epoch 1, Step 6000, loss: 0.06747680902481079, disc_loss: 0.001505840104073286 INFO:root:Epoch 1, Step 8000, loss: 0.04802854731678963, disc_loss: 0.014482944272458553 INFO:root:Epoch time: 5658.317040681839 INFO:root:Epoch 2, Step 2000, loss: 0.08109544962644577, disc_loss: 0.0069780051708221436 INFO:root:Epoch 2, Step 4000, loss: 0.07633058726787567, disc_loss: 0.006170968525111675 INFO:root:Epoch 2, Step 6000, loss: 0.08500031381845474, disc_loss: 0.0013521842192858458 INFO:root:Epoch 2, Step 8000, loss: 0.04127846658229828, disc_loss: 0.0018893849337473512 INFO:root:Epoch time: 5657.765841007233 INFO:root:Epoch 3, Step 2000, loss: 0.07090915739536285, disc_loss: 0.005887601524591446 INFO:root:Epoch 3, Step 4000, loss: 0.059316493570804596, disc_loss: 0.0012161008780822158 INFO:root:Epoch 3, Step 6000, loss: 0.10464320331811905, disc_loss: 0.0052590081468224525 INFO:root:Epoch 3, Step 8000, loss: 0.06500907987356186, disc_loss: 0.0010780195007100701 INFO:root:Epoch time: 5662.782829999924 INFO:root: pesq: 3.1471768543462115, csig: 4.524800468872296, cbak: 3.714835821897046, covl: 3.9118528691825403, ssnr: 9.58391283335014, stoi: 0.9514108003033614

INFO:root:Epoch 4, Step 2000, loss: 0.03907656669616699, disc_loss: 0.008593665435910225 INFO:root:Epoch 4, Step 4000, loss: 0.0602584108710289, disc_loss: 0.003996850922703743 INFO:root:Epoch 4, Step 6000, loss: 0.12089455872774124, disc_loss: 0.0021806589793413877 INFO:root:Epoch 4, Step 8000, loss: 0.09586295485496521, disc_loss: 0.0031526663806289434 INFO:root:Epoch time: 5669.503582715988 INFO:root:Epoch 5, Step 2000, loss: 0.0668821632862091, disc_loss: 0.0061484831385314465 INFO:root:Epoch 5, Step 4000, loss: 0.04616473987698555, disc_loss: 0.0014118528924882412 INFO:root:Epoch 5, Step 6000, loss: 0.06401266157627106, disc_loss: 0.0012459662975743413 INFO:root:Epoch 5, Step 8000, loss: 0.05762811750173569, disc_loss: 0.0017574067460373044 INFO:root:Epoch time: 5669.170885562897 INFO:root:Epoch 6, Step 2000, loss: 0.09387986361980438, disc_loss: 0.002971608191728592 INFO:root:Epoch 6, Step 4000, loss: 0.10805267840623856, disc_loss: 0.00345245492644608 INFO:root:Epoch 6, Step 6000, loss: 0.1099756583571434, disc_loss: 0.00208111759275198 INFO:root:Epoch 6, Step 8000, loss: 0.052734144032001495, disc_loss: 0.002511049387976527 INFO:root:Epoch time: 5673.420320510864"