Open kmanojkkmr opened 3 years ago
levanpon98
commented
3 years ago Try to run on GPU, this error happen when you use tf.gather_nd and the indices is out of bound index. When run on CPU, it will return an error, but when run on GPU, it will replace 0 in the corresponding output value
kmanojkkmr
commented
3 years ago Thanks for the response @levanpon98. I have changed "is_sampling_balanced" value from 1 to 0 in config.ini file. After the change training started and ran for 19 iterations and then got the same error. Currently we don't have an option to work on GPU systems. :( So, if there is any changes to be made like by replacing tf.gather_nd with other method(if any) please suggest.
Also do you have any idea how to inference on a single image and extract the information ? I have completed the table detection part. I should give the table's BBox coordinates for structure recognition and then extract the data from it.
Thanks a lot in advance.
Ankan1998
commented
3 years ago @kmanojkkmr is this issue solved for you. If yes can you please guide me about the solution. I am stuck with same error
ichko
commented
3 years ago @Ankan1998 @kmanojkkmr I had the same problem, it looks like this is a known issue in tf, and there is no solution:
This is a known issue in TF we cannot change without breaking backward compatibility; gather with invalid indices fails on the CPU but does not on the GPU :-/
https://github.com/tensorflow/tensorflow/issues/34788#issuecomment-561879705
Hello,
When i am training the model with the data which was mentioned in this repo(i.e. google drive link) I am getting below error.
I have tried this on Windows 8.1 CPU version and Linux 64 bit CPU. Tensorflow Version: 2.2.0 also tried on TF version: 1.15.3
I have modified the code according to 2.* version. But I couldn't find the fix for this issue. Could you please help me what changes to be done to overcome this.
Here is the traceback of the issue.
2020-09-01 10:26:56.662955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900] 2020-09-01 10:26:56.711510: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900] 2020-09-01 10:26:56.783215: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[21,788,3] = [21, 788, 900] does not index into param shape [25,900,900] 2020-09-01 10:26:58.902602: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12,450,3] = [12, 900] does not index into param shape [25,900,128] 2020-09-01 10:26:58.905858: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23,394,5] = [23, 900] does not index into param shape [25,900,128] 2020-09-01 10:26:58.912144: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[24,899,5] = [24, 900] does not index into param shape [25,900,128] Traceback (most recent call last): File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900] [[{{node conv_graph_dgcnn_fast_conv_1/GatherNd_10}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "bin/iterate/table_adjacency_parsing.py", line 35, in
trainer.train()
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../iterators/table_adjacency_parsing_iterator.py", line 84, in train
model.run_training_iteration(sess, summary_writer, iteration_number)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 481, in run_training_iteration
ops_result = sess.run(ops_to_run, feed_dict = feed_dict)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900]
[[node conv_graph_dgcnn_fast_conv_1/GatherNd_10 (defined at /home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py:259) ]]
Errors may have originated from an input operation. Input Source operations connected to node conv_graph_dgcnn_fast_conv_1/GatherNd_10: conv_graph_dgcnn_fast_conv_1/Placeholder_5 (defined at /home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py:163) conv_graph_dgcnn_fast_conv_1/concat_23 (defined at /home/sdpu ser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py:244)
Original stack trace for 'conv_graph_dgcnn_fast_conv_1/GatherNd_10': File "bin/iterate/table_adjacency_parsing.py", line 35, in
trainer.train()
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../iterators/table_adjacency_parsing_iterator.py", line 48, in train
model.initialize(training=True)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 96, in initialize
self.build_computation_graphs()
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 445, in build_computation_graphs
self.build_classification_segments(graph_features, placeholders)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 337, in build_classification_segments
sampled_indices, computation_graph, gt_matrix = self.do_monte_carlo_sampling(graph_features, gt_sampled_adj_matrix)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 259, in do_monte_carlo_sampling
return samples, x, tf.gather_nd(gt_matrix, indexing_tensor_for_adj_matrices)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, *kwargs)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 4852, in gather_nd_v2
return gather_nd(params, indices, name=name, batch_dims=batch_dims)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(args, **kwargs)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 4844, in gather_nd
return gen_array_ops.gather_nd(params, indices, name=name)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3666, in gather_nd
"GatherNd", params=params, indices=indices, name=name)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal
op_def=op_def)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1791, in init
self._traceback = tf_stack.extract_stack()