NVIDIA / DeepLearningExamples

State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
13.32k stars 3.19k forks source link

[SpeechSynthesis/Tactron2] Training crashes with AttributeError: module 'torch._C' has no attribute '_jit_set_autocast_mode' #1344

Open mikesol opened 1 year ago

mikesol commented 1 year ago

Related to Model/Framework(s) https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2

Describe the bug

The model cannot train because it hits the following error once in the docker image:

root@eff550f3bccb:/workspace/tacotron2# bash scripts/train_waveglow.sh
Traceback (most recent call last):
  File "train.py", line 42, in <module>
    import models
  File "/workspace/tacotron2/models.py", line 33, in <module>
    from waveglow.model import WaveGlow
  File "/workspace/tacotron2/waveglow/model.py", line 28, in <module>
    torch._C._jit_set_autocast_mode(False)
AttributeError: module 'torch._C' has no attribute '_jit_set_autocast_mode'
Traceback (most recent call last):
  File "train.py", line 42, in <module>
    import models
  File "/workspace/tacotron2/models.py", line 33, in <module>
    from waveglow.model import WaveGlow
  File "/workspace/tacotron2/waveglow/model.py", line 28, in <module>
    torch._C._jit_set_autocast_mode(False)
AttributeError: module 'torch._C' has no attribute '_jit_set_autocast_mode'
root@eff550f3bccb:/workspace/tacotron2# exit

To Reproduce Steps to reproduce the behavior:

  1. Follow the readme instructions verbatim for training waveglow. The failure happens on bash scripts/train_waveglow.sh.

Environment Please provide at least:

sp1cae commented 2 months ago

same error