Open ckcz123 opened 7 years ago
And I wonder why it need 7.23GiB memory? Can anyone explain it?
I think your seq length is very high. 1200 is quite long. Tensorflow has this issue with the way it creates these kinds of graphs using seq2seq see this issue. You may try to remake it using dynamic_rnn as they suggest in the comments. A quick fix might be to lower your batch size. Though it is already low so your loss may be noiser.
@fujimotomh Thank you for your reply. But I wonder how can I modify the code as I'm just new to tensorflow?
I just tried to use
outputs, last_state = tf.nn.rnn(cell, inputs, initial_state=self.initial_state, scope='rnnlm')
instead of
outputs, last_state = seq2seq.rnn_decoder(inputs, self.initial_state, cell, loop_function=loop if infer else None, scope='rnnlm')
, and it can work! (But I don't know if the result is correct).
But if I tried
outputs, last_state = tf.nn.dynamic_rnn(cell, inputs, initial_state=self.initial_state, scope='rnnlm')
, it would throwValueError: Dimension must be 2 but is 3
.
So how can I modify the code?
@ckcz123 You almost have it. dynamic_rnn
takes the input as a tensor and not a list. This works on my laptop with a seq_length of 1200.
outputs, last_state = tf.nn.dynamic_rnn(cell, tf.nn.embedding_lookup(embedding, self.input_data), initial_state=self.initial_state, scope='rnnlm')
To confirm correctness, I think the best thing to do would be to run it with default parameters and see if you can get low loss on the training set. I would suspect this would work though as rnn_decoder
and dynamic_rnn
claim have the same function.
@fujimotomh Oh, it works! Only 1.1G usage of GPU memory! Thanks for your advice!
What a wonderful project! I have used it to solve some problems. But there is one problem that always bothers me.
In one of the cases, I have to use
rnn_size=512
,num_layers=2
,seq_length=1200
. Other arguments:batch_size=10
,num_epochs=50
,grad_clip=5.0
, and so on. But it will allocate 7.23GiB in GPU, which is only 8GB-free. So I just wonder if I can reduce GPU memory to 7GiB or less. If so, I can run it on GPU.rnn_size
,num_layers
,seq_length
cannot be modified.Here is some of the ouputs.
Sorry for my poor English, and thanks a lot!