search

NVIDIA / radtts

Provides training, inference and voice conversion recipes for RADTTS and RADTTS++: Flow-based TTS models with Robust Alignment Learning, Diverse Synthesis, and Generative Modeling and Fine-Grained Control over of Low Dimensional (F0 and Energy) Speech Attributes.
MIT License
280 stars 40 forks source link

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn #18

Closed egorsmkv closed 1 year ago

egorsmkv commented 1 year ago

I have successfully trained a model on the first step (with the decoder config), but training fails on the second step when I am trying to train a model with config_ljs_agap.json config.

The error:

...
setting up tboard log in outdir_model/logs
Training voiced prediction
Training voiced embeddings
Epoch: 0
/home/yehor/.local/lib/python3.8/site-packages/torch/nn/modules/rnn.py:777: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at ../aten/src/ATen/native/cudnn/RNN.cpp:968.)
  result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
/home/yehor/.local/lib/python3.8/site-packages/torch/nn/modules/rnn.py:777: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at ../aten/src/ATen/native/cudnn/RNN.cpp:968.)
  result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
Traceback (most recent call last):
  File "train.py", line 499, in <module>
    train(n_gpus, rank, **train_config)
  File "train.py", line 417, in train
    scaler.scale(loss).backward()
  File "/home/yehor/.local/lib/python3.8/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/home/yehor/.local/lib/python3.8/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

A fork of the source code is here: https://github.com/egorsmkv/radtts/

I am changed the code to support Ukrainian.

Environment:

egorsmkv commented 1 year ago

It's my fault, I have added the model_config.include_modules="decatndur" argument during training