search

TensorSpeech / TensorFlowTTS

:stuck_out_tongue_closed_eyes: TensorFlowTTS: Real-Time State-of-the-art Speech Synthesis for Tensorflow 2 (supported including English, French, Korean, Chinese, German and Easy to adapt for other languages)
https://tensorspeech.github.io/TensorFlowTTS/
Apache License 2.0
3.84k stars 815 forks source link

Getting error when training tacotron-2 from scratch #218

Closed sabby993 closed 4 years ago

sabby993 commented 4 years ago

Hi there. I am new to tensorflow and text to speech systems in general and I was following your example of training tacotron-2 from scratch and I ran into the following error when I am running train_tacotron2.py as follows:

!CUDA_VISIBLE_DEVICES=0 python examples/tacotron2/train_tacotron2.py \ --train-dir ./dump/train/ \ --dev-dir ./dump/valid/ \ --outdir ./examples/tacotron2/exp/train.tacotron2.v1/ \ --config ./examples/tacotron2/conf/tacotron2.v1.yaml \ --use-norm 1 \ --mixed_precision 0 \ --resume ""

I am getting the following error:

2020-08-19 06:51:37.897818: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-08-19 06:51:39.335464: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2020-08-19 06:51:39.369204: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:39.369813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5 coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.73GiB deviceMemoryBandwidth: 298.08GiB/s 2020-08-19 06:51:39.369857: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-08-19 06:51:39.371648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-08-19 06:51:39.373501: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2020-08-19 06:51:39.373898: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2020-08-19 06:51:39.376217: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2020-08-19 06:51:39.377691: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2020-08-19 06:51:39.382127: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-08-19 06:51:39.382245: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:39.382891: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:39.383447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 [nltk_data] Downloading package averaged_perceptron_tagger to [nltk_data] /root/nltk_data... [nltk_data] Unzipping taggers/averaged_perceptron_tagger.zip. [nltk_data] Downloading package cmudict to /root/nltk_data... [nltk_data] Unzipping corpora/cmudict.zip. 2020-08-19 06:51:43.531632: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-08-19 06:51:43.536795: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2200000000 Hz 2020-08-19 06:51:43.537056: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4a62fc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-08-19 06:51:43.537102: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-08-19 06:51:43.672860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:43.673576: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4a628c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-08-19 06:51:43.673611: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla T4, Compute Capability 7.5 2020-08-19 06:51:43.674804: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:43.675440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5 coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.73GiB deviceMemoryBandwidth: 298.08GiB/s 2020-08-19 06:51:43.675484: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-08-19 06:51:43.675540: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-08-19 06:51:43.675563: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2020-08-19 06:51:43.675584: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2020-08-19 06:51:43.675606: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2020-08-19 06:51:43.675627: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2020-08-19 06:51:43.675649: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-08-19 06:51:43.675738: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:43.676390: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:43.676948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-08-19 06:51:43.677012: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-08-19 06:51:44.454742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-08-19 06:51:44.454799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 2020-08-19 06:51:44.454812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N 2020-08-19 06:51:44.454999: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:44.455639: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-08-19 06:51:44.456208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13936 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5) Traceback (most recent call last): File "examples/tacotron2/train_tacotron2.py", line 473, in main() File "examples/tacotron2/train_tacotron2.py", line 379, in main use_fixed_shapes=config["use_fixed_shapes"], File "/content/drive/My Drive/projectFiles/TensorFlowTTS/examples/tacotron2/tacotron_dataset.py", line 93, in init assert len(mel_files) != 0, f"Not found any mels files in ${root_dir}." AssertionError: Not found any mels files in $./dump/train/.

I am running all my code on google colab and I completed all the preprocessing steps before starting the training. Can you please guide me through this error.

dathudeptrai commented 4 years ago

Hi thể, ./dump/train is just example, you need change to ur dir when u preprocessing, for example ./dump_ljspeech/train

sabby993 commented 4 years ago

Thanks a lot. Got it working now.