train.py doesn't run properly

[Nathan-Chell]

Im running Windows 10, using the code used in the Sentdex tutorial. I'm using tensorflow 1.14rc1. I have prepared data, basically followed along with the tutorial. When running train.py with CMD inside nmt-chatbot dir, the GPU is recognized however, I get a whole string of errors after about 25 seconds of running.

`0 successful operations. 0 derived errors ignored.

Original stack trace for 'dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul': File "train.py", line 18, in tf.app.run(main=nmt.main, argv=[os.getcwd() + '\nmt\nmt\nmt.py'] + unparsed) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 300, in run _run_main(main, args) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 251, in _run_main sys.exit(main(argv)) File "Q:\nmt-chatbot/nmt\nmt\nmt.py", line 539, in main run_main(FLAGS, default_hparams, train_fn, inference_fn) File "Q:\nmt-chatbot/nmt\nmt\nmt.py", line 532, in run_main train_fn(hparams, target_session=target_session) File "Q:\nmt-chatbot/nmt\nmt\train.py", line 223, in train eval_model = model_helper.create_eval_model(model_creator, hparams, scope) File "Q:\nmt-chatbot/nmt\nmt\model_helper.py", line 157, in create_eval_model extra_args=extra_args) File "Q:\nmt-chatbot/nmt\nmt\attention_model.py", line 61, in init extra_args=extra_args) File "Q:\nmt-chatbot/nmt\nmt\model.py", line 97, in init res = self.build_graph(hparams, scope=scope) File "Q:\nmt-chatbot/nmt\nmt\model.py", line 284, in build_graph encoder_outputs, encoder_state, hparams) File "Q:\nmt-chatbot/nmt\nmt\model.py", line 407, in _build_decoder logits = self.output_layer(outputs.rnn_output) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\layers\base.py", line 537, in call outputs = super(Layer, self).call(inputs, *args, kwargs) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 634, in call outputs = call_fn(inputs, *args, *kwargs) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 146, in wrapper ), args, kwargs) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 450, in converted_call result = converted_f(effective_args, kwargs) File "C:\Users\nathp\AppData\Local\Temp\tmpdem_lx0b.py", line 63, in tfcall outputs = ag.if_stmt(cond_2, if_true_2, if_false_2, get_state_2, set_state_2) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 441, in if_stmt return _py_if_stmt(cond, body, orelse) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 528, in _py_if_stmt return body() if cond else orelse() File "C:\Users\nathp\AppData\Local\Temp\tmpdem_lx0b.py", line 23, in if_true_2 outputs = ag.converted_call('tensordot', standard_ops, ag.ConversionOptions(recursive=True, force_conversion=False, optional_features=(), internal_convert_user_code=True), (inputs, self.kernel, [[rank - 1], [0]]), None) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 356, in converted_call return _call_unconverted(f, args, kwargs) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 255, in _call_unconverted return f(args) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\ops\math_ops.py", line 3799, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\util\dispatch.py", line 180, in wrapper return target(args, *kwargs) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2647, in matmul a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 6295, in mat_mul name=name) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func return func(args, **kwargs) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op op_def=op_def) File "C:\Users\nathp\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in init self._traceback = tf_stack.extract_stack()`

[Nathan-Chell]

Okay, now im trying the train.py file which was inside nmt-chatbot/nmt/nmt.

I now get the error: ImportError: cannot import name 'attention_model' from 'main' (train.py)

[Nathan-Chell]

When 'from .' is removed, then i get a no module named 'attention_model' error.

[Nathan-Chell]

Fixed:

The problem was down to a 'ResourceExhausstedError' from tensorflow, Basically whatever your vocab size is * your num_units. If you get this error then you have either too high of a vocab size and time many units, so decrease one(vocab size would be better).

In my case, i had a vocab size of 150003 and 512 units and I got this error. I have an rtx2070 with 8GB of memory.