search

suriyadeepan / practical_seq2seq

A simple, minimal wrapper for tensorflow's seq2seq module, for experimenting with datasets rapidly
http://suriyadeepan.github.io/2016-12-31-practical-seq2seq/
GNU General Public License v3.0
569 stars 270 forks source link

OOM when allocating tensor with shape[2048,4096] #28

Open felansu opened 7 years ago

felansu commented 7 years ago

When i try to run python 04-Cornell-Movie-Dialog-Bot.py get this error:

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000079CFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000079EFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A0FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A2FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A4FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A6FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A8FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007AAFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007ACFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007AEFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B0FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B2FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B4FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B6FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B8FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007BAFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007BCFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007BEFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007C0FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007C2FE2100 of size 36160768
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710DC3600 of size 8448
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710DC5800 of size 5120
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710DC6D00 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710E03700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710EE3800 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710F23900 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710F63C00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000711143D00 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007111A3D00 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007112C3D00 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F51C600 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F6FC600 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F798600 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F7BC600 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F8BC600 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F91C600 of size 917504
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FA1C600 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FB40600 of size 901120
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FC3C600 of size 3252480
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FF96700 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740076700 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007400B6700 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740196700 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074023A700 of size 3236096
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740570800 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740634800 of size 1818624
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007407F4800 of size 2449664
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740A6A900 of size 1179648
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740BAA900 of size 5480704
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000741104A00 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007411C4A00 of size 3907840
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074159EB00 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074165EB00 of size 393216
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007416DEB00 of size 393216
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074175EB00 of size 3121408
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000741A78C00 of size 5480704
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000741FF2D00 of size 1835008
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007421D2D00 of size 4038912
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:693]      Summary of in-use Chunks by size:
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 25 Chunks of size 256 totalling 6.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 6400 totalling 6.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 93 Chunks of size 16384 totalling 1.45MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 5 Chunks of size 32256 totalling 157.5KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 678 Chunks of size 131072 totalling 84.75MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 241152 totalling 235.5KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 257792 totalling 251.8KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 51 Chunks of size 262144 totalling 12.75MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 35 Chunks of size 393216 totalling 13.13MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 638976 totalling 624.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 7 Chunks of size 32776192 totalling 218.80MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 81 Chunks of size 33554432 totalling 2.53GiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 35646464 totalling 34.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 36160768 totalling 34.49MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 42034944 totalling 40.09MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 66330624 totalling 63.26MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:700] Sum Total of in-use chunks: 3.02GiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:702] Stats:
Limit:                  3292128870
InUse:                  3246350848
MaxInUse:               3249725952
NumAllocs:                    7135
MaxAllocSize:             66330624

W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:274] **************************************************************************
**************************
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:275] Ran out of memory trying to allocate 32.00MiB.  See logs for memory state.
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[2048,4096]
Traceback (most recent call last):
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _do_call
    return fn(*args)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1003, in _run_fn
    status, run_metadata)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,4096]
         [[Node: gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=true,
 transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/concat, gradients
/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/add_grad/tuple/control_dependency)]]
         [[Node: Adam/update_decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam/_410 = _Recv[client_terminated=false, recv_d
evice="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_656_Adam/update_decoder/embedding_rnn_seq2seq/e
mbedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "04-Cornell-Movie-Dialog-Bot.py", line 42, in <module>
    sess = model.train(train_batch_gen, val_batch_gen)
  File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 144, in train
    self.train_batch(sess, train_set)
  File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 102, in train_batch
    _, loss_v = sess.run([self.train_op, self.loss], feed_dict)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 766, in run
    run_metadata_ptr)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,4096]
         [[Node: gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=true,
 transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/concat, gradients
/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/add_grad/tuple/control_dependency)]]
         [[Node: Adam/update_decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam/_410 = _Recv[client_terminated=false, recv_d
evice="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_656_Adam/update_decoder/embedding_rnn_seq2seq/e
mbedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op 'gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1', defined at:
  File "04-Cornell-Movie-Dialog-Bot.py", line 30, in <module>
    num_layers=3
  File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 79, in __init__
    __graph__()
  File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 75, in __graph__
    self.train_op = tf.train.AdamOptimizer(learning_rate=lr).minimize(self.loss)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\optimizer.py", line 269, in minimize
    grad_loss=grad_loss)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\optimizer.py", line 335, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 482, in gradients
    in_grads = grad_fn(op, *out_grads)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_grad.py", line 731, in _MatMulGrad
    math_ops.matmul(op.inputs[0], grad, transpose_a=True))
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1729, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1442, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

...which was originally created as op 'decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul', defined at:
  File "04-Cornell-Movie-Dialog-Bot.py", line 30, in <module>
    num_layers=3
[elided 0 identical lines from previous traceback]
  File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 79, in __init__
    __graph__()
  File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 58, in __graph__
    xvocab_size, yvocab_size, emb_dim)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\seq2seq.py", line 357, in embedding_rnn_seq2seq
    feed_previous=feed_previous)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\seq2seq.py", line 280, in embedding_rnn_decoder
    loop_function=loop_function)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\seq2seq.py", line 146, in rnn_decoder
    output, state = cell(inp, state)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 586, in __call__
    output, res_state = self._cell(inputs, state)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 815, in __call__
    cur_inp, new_state = cell(cur_inp, cur_state)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 684, in __call__
    output, new_state = self._cell(inputs, state, scope)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 311, in __call__
    concat = _linear([inputs, h], 4 * self._num_units, True)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 909, in _linear
    res = math_ops.matmul(array_ops.concat(1, args), matrix)
  File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1729, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2048,4096]
         [[Node: gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=true,
 transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/concat, gradients
/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/add_grad/tuple/control_dependency)]]
         [[Node: Adam/update_decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam/_410 = _Recv[client_terminated=false, recv_d
evice="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_656_Adam/update_decoder/embedding_rnn_seq2seq/e
mbedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_gpu_executor.cc:652] Deallocating stream with pending work
qinglingni commented 7 years ago

I keep getting the same error too. Running this on Google cloud instance with 2 GPU on a train dataset with only 40K data. Was wondering if anyone found the issue or solution for this problem.