State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
13.32k
stars
3.19k
forks
source link
[SpeechSynthesis/Tactron2] Training crashes with AttributeError: module 'torch._C' has no attribute '_jit_set_autocast_mode' #1344
The model cannot train because it hits the following error once in the docker image:
root@eff550f3bccb:/workspace/tacotron2# bash scripts/train_waveglow.sh
Traceback (most recent call last):
File "train.py", line 42, in <module>
import models
File "/workspace/tacotron2/models.py", line 33, in <module>
from waveglow.model import WaveGlow
File "/workspace/tacotron2/waveglow/model.py", line 28, in <module>
torch._C._jit_set_autocast_mode(False)
AttributeError: module 'torch._C' has no attribute '_jit_set_autocast_mode'
Traceback (most recent call last):
File "train.py", line 42, in <module>
import models
File "/workspace/tacotron2/models.py", line 33, in <module>
from waveglow.model import WaveGlow
File "/workspace/tacotron2/waveglow/model.py", line 28, in <module>
torch._C._jit_set_autocast_mode(False)
AttributeError: module 'torch._C' has no attribute '_jit_set_autocast_mode'
root@eff550f3bccb:/workspace/tacotron2# exit
To Reproduce
Steps to reproduce the behavior:
Follow the readme instructions verbatim for training waveglow. The failure happens on bash scripts/train_waveglow.sh.
Environment
Please provide at least:
Container version (e.g. pytorch:19.05-py3): NVIDIA Release 20.06 (build 13419386), PyTorch Version 1.6.0a0+9907a3e
GPUs in the system: 2x 1080TI
CUDA driver version (e.g. 418.67): Cuda compilation tools, release 11.6, V11.6.124, Build cuda_11.6.r11.6/compiler.31057947_0
Related to Model/Framework(s) https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2
Describe the bug
The model cannot train because it hits the following error once in the docker image:
To Reproduce Steps to reproduce the behavior:
bash scripts/train_waveglow.sh
.Environment Please provide at least: