I have problem while I'm tranning.

[hungptit123]

Training Iteration 105: Accuracy - cells: 0.978424609 rows: 0.895143926 cols: 0.803354 Loss - cells: 0.0991929844 rows: 0.293497771 cols: 0.486667097 Fraction - cells: 0.0215753876 rows: 0.104856044 cols: 0.19664596 Total loss: 0.293119282 Training Iteration 106: Accuracy - cells: 0.98035419 rows: 0.896977782 cols: 0.827894092 Loss - cells: 0.0928051174 rows: 0.2944884 cols: 0.456886739 Fraction - cells: 0.0196458325 rows: 0.103022188 cols: 0.172105893 Total loss: 0.281393409 Training Iteration 107: 2019-12-02 11:23:50.896536: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26,190] = [26, 109, 90] does not index into param shape [30,109,109,128] Traceback (most recent call last): File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call return fn(*args) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[26,190] = [26, 109, 90] does not index into param shape [30,109,109,128] [[{{node conv_graph_dgcnn_fast_conv_1/GatherNd}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "table_adjacency_parsing.py", line 31, in trainer.train() File "/home/hungdv/Documents/graphNN/TIES-verions2/python/iterators/table_adjacency_parsing_iterator.py", line 88, in train model.run_training_iteration(sess, summary_writer, iteration_number) File "/home/hungdv/Documents/graphNN/TIES-verions2/python/models/basic_model.py", line 400, in run_training_iteration ops_result = sess.run(ops_to_run, feed_dict = feed_dict) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run run_metadata) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[26,190] = [26, 109, 90] does not index into param shape [30,109,109,128] [[node conv_graph_dgcnn_fast_conv_1/GatherNd (defined at /home/hungdv/Documents/graphNN/TIES-verions2/python/ops/ties.py:32) ]]

Errors may have originated from an input operation. Input Source operations connected to node conv_graph_dgcnn_fast_conv_1/GatherNd: conv_graph_dgcnn_fast_conv_1/Cast (defined at /home/hungdv/Documents/graphNN/TIES-verions2/python/ops/ties.py:31)
conv_graph_dgcnn_fast_conv_1/conv2d_14/LeakyRelu (defined at /tmp/tmp8vnsy1dh.py:74)

Original stack trace for 'conv_graph_dgcnn_fast_conv_1/GatherNd': File "table_adjacency_parsing.py", line 31, in trainer.train() File "/home/hungdv/Documents/graphNN/TIES-verions2/python/iterators/table_adjacency_parsing_iterator.py", line 53, in train model.initialize(training=True) File "/home/hungdv/Documents/graphNN/TIES-verions2/python/models/basic_model.py", line 94, in initialize self.build_computation_graphs() File "/home/hungdv/Documents/graphNN/TIES-verions2/python/models/basic_model.py", line 365, in build_computation_graphs vertices_y2, vertices_x2, scale_y, scale_x) File "/home/hungdv/Documents/graphNN/TIES-verions2/python/ops/ties.py", line 32, in gather_features_from_conv_head return tf.gather_nd(conv_head, indexing_tensor) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper return target(*args, *kwargs) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 3796, in gather_nd return gen_array_ops.gather_nd(params, indices, name=name) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3991, in gather_nd "GatherNd", params=params, indices=indices, name=name) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(args, **kwargs) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op op_def=op_def) File "/home/hungdv/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init self._traceback = tf_stack.extract_stack()

[kmanojkkmr]

Hello @hungptit123,

I am facing the same issue after completing 19 iterations. Could you please help me how you solved this issue?

Thank you in advance for you help. :)