search

BenAAndrew / Voice-Cloning-App

A Python/Pytorch app for easily synthesising human voices
BSD 3-Clause "New" or "Revised" License
1.4k stars 233 forks source link

Low GPU usage #104

Closed AnasFX closed 2 years ago

AnasFX commented 2 years ago

Hi, thanks for this awesome work. Is it normal that the GPU usage is not full all the time, it is fluctuating always between 0% - 50% - 30% - 40%. Is it bug or a normal thing, because the speed would be much much more faster and efficient if the GPU usage was 100% all the time. If it is normal, would you kindly explain why is this happening please.

Specs : Win11 - 3090 - Latest build of Voice - Cloning-App - the data set that has been used is : David Attenborough

One more question please : is it possible to stop during training without losing the model, because during training there is no button to stop training and save the model then exit. So I have to close it forcefully and with that the progress will be lost.

Thanks a lot. Best regards.

BenAAndrew commented 2 years ago

Hi @AnasFX,

  1. I'm assuming you're talking about GPU usage during training in the app. To increase GPU usage you just need to increase batch size under the advanced options of training. You should aim for ~90% utilization but remember that setting this too high will cause an error.

  2. Checkpoints are saved frequently so that your progress is not lost if you stop the app or it crashes (default is every 1000 iterations but this can be changed in advanced options of training). You can select this checkpoint to carry on from on the training page

AnasFX commented 2 years ago

Thanks a lot for you reply. Unfortunately this didn't solve the low GPU training usage problem. I do believe this is a problem with tacotron2. I read a similar question to mine on their GitHub issues, and no one answered him.