search

NVIDIA / OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
https://nvidia.github.io/OpenSeq2Seq
Apache License 2.0
1.54k stars 372 forks source link

How much time take tacotron gst infererence?/infererence stuck #489

Open hadaev8 opened 4 years ago

hadaev8 commented 4 years ago

It stuck on

Successfully opened dynamic library libcublas.so.10.0

Code https://colab.research.google.com/drive/10DxRZTqmIPE6983fGEZ2UDjRrcvM99pK

blisc commented 4 years ago

Can you confirm that the code is running on GPU and not CPU?

hadaev8 commented 4 years ago

Should be gpu, but actually i'm not sure

blisc commented 4 years ago

Tacotron gst is known to hang on CPU #439. Although another user has reported errors on Google colab #476. Unfortunately, just like the CPU issue, we most likely will not be able to fix it if it is a Google colab issue.

JarvisStark007 commented 4 years ago

Same issue. It is running on local single GPU(1660 Ti) in mine and it is definitely using a GPU(nvidia_smi shows it's being used). I'm training a tacatron_float for text-to-speech conversion model on the LJSpeech dataset. I have tried reducing batch size but it doesn't seem to fix the issue.

smdshakeelhassan commented 4 years ago

Were you able to resolve the issue?I am also facing the same issue. I am using 6 GB Nvidia Geforce GTX 1060 with Cuda 10.0 and nvidia drivers 430.64. My batch size is 32. I had to reduce it from 48 due to cuda out of memory error. GPU utilisation in Nvidia-smi shows 1988/6078 MiB utilised. Volatile GPU util is very less (1%-4%)

conqueror7 commented 3 years ago

Use tf.GraphKeys.GLOBAL_VARIABLES instead. *** WARNING: Can't compute number of objects per step, since train model does not define get_num_objects_per_step method. 2020-08-04 14:37:19.226539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0

I am also facing the same issue. Any solution for the same? I am using tacotron_gst.py config file and pre-trained weights shared here (https://nvidia.github.io/OpenSeq2Seq/html/speech-synthesis.html). Other config is shared below- GPU- V100 Cuda - 10.0 Python - 3.6.10 Tensorflow - 1.15.0