Closed laucheukhim closed 3 years ago
That example is not using Keras CuDNNLSTM layers (introduced in 2.0.9)? https://keras.io/layers/recurrent/#cudnnlstm https://github.com/keras-team/keras/releases
@hiandersson I modified it, just to test, please read carefully. There are 2 separate tests I run. One with the original script, another one with LSTM
replaced by CuDNNLSTM
. Both tests indicate that TITAN V runs slower.
@laucheukhim read too fast (just looked at the example), my bad! Its interesting that CuDNNLSTM on TitanXp is 26 x faster than 'normal' LSTM. Is this the best overall performance you can get (comparing with Theano, CNTK)?
@hiandersson Yes. Unfortunately, CuDNNLSTM does not support dropout
and recurrent_dropout
, so it cannot be used as a drop in replacement.
@laucheukhim is his a problem in Keras, tensorflow or with nvidia? is there any prognosis for a fix?
@hiandersson I want a fix too. TITAN V is too expensive to perform badly like this.
I don't know what the cause is. I see similar degradation in pytorch too. This could be due to the lack of cuDNN autotune.
TITAN V runs 15% faster in games, 75% faster in lyra2rev2 crypto mining, but 60% slower in my LSTM model. This is a big disappointment for what is meant to be a machine learning card.
Just for curiosity, have you tried CNTK with CuDNNLSTM ? How was that?
@grafael CuDNNLSTM is only available on the TensorFlow backend.
Thanks @laucheukhim
LSTM training on Keras is significantly slower with TITAN V than that with TITAN Xp. I ran the IMDB Bidirectional LSTM example on multiple backends and found that TITAN V is at best 4% faster and at worst 147% slower than TITAN Xp.
OS: Ubuntu 16.04.3 LTS Drivers: NVIDIA 387.34, CUDA 9.1, cuDNN 7.0.5 Python: 3.6 Anaconda
I ran the example on multiple backends, and I also ran a modified one that uses CuDNNLSTM on Tensorflow. The following results are in seconds per epoch:
This looks like a continuation of the problem in #6640 that LSTM performance decreases with generation.