Training is very slow

[kevaday]

Hi Skuldur,

I am using your training script to train with my own midi files. I don't know why, but the training is going very slowly. It takes about 25 minutes to train only one epoch. I'm confused because i have a brand new GTX 1070 with drivers installed including geforce experience, cuda 9.0 (because tensorflow is incompatible with version 9.2 for some reason), and cudNN. With another training script (it's for generating text) it took only 1-2 minutes per epoch, but its model only had 1 LSTM so that might also be relatively slow. How long did it take you to train with your midi files, with what GPU? If you can help, it would be greatly appreciated.

Thanks, Kevaday

[kevaday]

Also, I tried using your text generating LSTM but it's also very slow on my GPU, it takes about 1 hour to do one epoch. Shouldn't my GPU be a lot faster at training?

[mehrzeller]

you can use CuDNNLSTM instead : from keras.layers import CuDNNLSTM instead of from keras.layers import LSTM don't forget to change all your LSTM statements in the source code

[kevaday]

Thanks mehrzeller, actually, it's a coincidence because I was just looking at the keras recurrent layers site and just saw that there's such thing but thanks anyways

[victor-felicitas]

For new Keras versions specifying CuDNNLSTM is obsolete. Training is slow because with recurrent_dropout >0 CUDNN can not be used.

From Keras documentation: _"The requirements to use the cuDNN implementation are: activation == tanh recurrent_activation == sigmoid recurrent_dropout == 0 unroll is False usebias is True Inputs, if use masking, are strictly right-padded. Eager execution is enabled in the outermost context."

I suggest changing the model by adding dropout layer instead, after each LSTM layer