I encountered a problem while training the lucidrains_dvae model. Initially, the loss of the model seemed normal, but after about 5k steps, the model started to crash. After the crash, the model was unable to correctly reconstruct the mel spectrogram and instead generated a bunch of noise. I used the dvae configuration file that you provided in a previous discussion and trained it on my personal dataset. I have tried multiple different datasets, but the problem persists. I suspect that it is not an issue with the dataset itself. I have also tried different learning rates and batch sizes (from 128 to 12288), but the problem still occurs. I have attached the plot of my loss curve.
Have you encountered such a problem when training the dvae model, how did you solve it, thank you for your answer.
I would greatly appreciate your help in resolving this issue. Thank you very much for your patience in answering.
Thank you for your excellent work - Tortoise !!!
I encountered a problem while training the lucidrains_dvae model. Initially, the loss of the model seemed normal, but after about 5k steps, the model started to crash. After the crash, the model was unable to correctly reconstruct the mel spectrogram and instead generated a bunch of noise. I used the dvae configuration file that you provided in a previous discussion and trained it on my personal dataset. I have tried multiple different datasets, but the problem persists. I suspect that it is not an issue with the dataset itself. I have also tried different learning rates and batch sizes (from 128 to 12288), but the problem still occurs. I have attached the plot of my loss curve.
Have you encountered such a problem when training the dvae model, how did you solve it, thank you for your answer.
I would greatly appreciate your help in resolving this issue. Thank you very much for your patience in answering.