yl4579 / StyleTTS2

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
MIT License
4.97k stars 419 forks source link

Getting error in d_loss.backward() of first_stage training #260

Open SandyPanda-MLDL opened 5 months ago

SandyPanda-MLDL commented 5 months ago

I am getting error in the following part of the first training stage code:

discriminator loss

        if epoch >= TMA_epoch:
            torch.autograd.set_detect_anomaly(True)
            optimizer.zero_grad()
            d_loss = dl(wav.detach().clone().unsqueeze(1).float(), y_rec.detach().clone()).mean()
            #d_loss = dl(wav.detach().unsqueeze(1).float(), y_rec.detach()).mean()
            #accelerator.backward(d_loss)
            print(f'Discriminator loss is {d_loss} and requires_grad is {d_loss.requires_grad} and epoch is {epoch}')
            d_loss.backward()
            optimizer.step('msd')
            optimizer.step('mpd')
            print(f'Discriminator loss is {d_loss} and requires_grad is {d_loss.requires_grad} and epoch is {epoch}')
        else:
            d_loss = 0
            print(f'Else discriminator loss is {d_loss} and epoch is {epoch}')

The error is: File "train_first.py", line 293, in main d_loss.backward() File "/hdd5/Sandipan/envs/styletts1/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward self, gradient, retain_graph, create_graph, inputs=inputs File "/hdd5/Sandipan/envs/styletts1/lib/python3.7/site-packages/torch/autograd/init.py", line 199, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2, 1, 6000, 5]] is at version 6; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!