search

NVIDIA / OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
https://nvidia.github.io/OpenSeq2Seq
Apache License 2.0
1.54k stars 371 forks source link

Wave2Letter+ #437

Closed flassTer closed 5 years ago

flassTer commented 5 years ago

Hello, I have came up with a problem while training Wave2Letter+ on the LibriSpeech dataset from scratch. The terminal displays:

WARNING: tensorflow: It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 10835 vs. previous value: 10835. You could increase the global step by passing tf.train.get_global_step() to Optimiser.apply_gradients or Optimiser.minimize.

This has happened twice in a row. The first time I thought something may have gone wrong so I restarted the training and when it reached the same step again, it displayed the same error. The training is ran on 4 GPUs Tesla K80.

borisgin commented 5 years ago

This is related to mixed precision. K80 does not support float16, so you’d better switch to float.

flassTer commented 5 years ago

Thank you @borisgin

rusiaaman commented 5 years ago

@borisgin I am facing the same issue but on single Tesla T4. Any pointers?