search

shahrukhqasim / TIES-2.0

Code for: S.R. Qasim, H. Mahmood, and F. Shafait, Rethinking Table Recognition using Graph Neural Networks (2019)
MIT License
276 stars 69 forks source link

InvalidArgumentError: Input to reshape is a tensor with 810000 values, but the requested shape has 27000 [Op:Reshape] #37

Closed kmanojkkmr closed 4 years ago

kmanojkkmr commented 4 years ago

Hi Team,

When I am trying to run the training part(I am using the partial dataset which was provided by you in ReadMe) , I am getting the below error.

Max vertices: 900 Max word length: 30

vertex_text: tf.Tensor( [[ 82 69 76 ... 0 0 0] [ 76 79 78 ... 0 0 0] [109 103 97 ... 0 0 0] ... [ 68 105 115 ... 0 0 0] [ 79 70 0 ... 0 0 0] [ 74 97 110 ... 0 0 0]], shape=(30, 27000), dtype=int64) shape of vertex_text: (30, 27000) Traceback (most recent call last): File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8075, in reshape tld.op_callbacks, tensor, shape) tensorflow.python.eager.core._FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "bin/iterate/table_adjacency_parsing.py", line 35, in trainer.train() File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python/bin/iterate/../../iterators/table_adjacency_parsing_iterator.py", line 48, in train model.initialize(training=True) File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python/bin/iterate/../../models/basic_model.py", line 72, in initialize self.training_feeds = self.training_reader.get_feeds() File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python/bin/iterate/../../readers/image_words_reader.py", line 76, in get_feeds vertex_text = tf.reshape(vertex_text, shape=(self.num_max_vertices, self.max_word_length)) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 193, in reshape result = gen_array_ops.reshape(tensor, shape, name) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8080, in reshape tensor, shape, name=name, ctx=_ctx) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8107, in reshape_eager_fallback ctx=ctx, name=name) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 810000 values, but the requested shape has 27000 [Op:Reshape]

Here it is expecting a tensor of shape 810000 as the shape of vertex_text is (30, 27000).

To overcome this I have modified the parameter max_vertices from 900 to 27000

After this change, I am getting other error:

File "bin/iterate/table_adjacency_parsing.py", line 35, in trainer.train() File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python/bin/iterate/../../iterators/table_adjacency_parsing_iterator.py", line 48, in train model.initialize(training=True) File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python/bin/iterate/../../models/basic_model.py", line 72, in initialize self.training_feeds = self.training_reader.get_feeds() File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python/bin/iterate/../../readers/image_words_reader.py", line 71, in get_feeds vertex_features, vertex_text, image, global_features, adj_cells, adj_rows, adj_cols = iterator.get_next() File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 741, in get_next return self._next_internal() File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 661, in _next_internal return structure.from_compatible_tensor_list(self._element_spec, ret) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/contextlib.py", line 99, in exit self.gen.throw(type, value, traceback) File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 1989, in execution_mode executor_new.wait() File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/eager/executor.py", line 67, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.InvalidArgumentError: Key: adjacency_matrix_cols. Can't parse serialized Example. [[{{node ParseSingleExample/ParseExample/ParseExampleV2}}]]

Can you please look into this and help me what can be done.

Thank you in Advance.

kmanojkkmr commented 4 years ago

I have resolved this issue by changing the code in https://github.com/shahrukhqasim/TIES-2.0/blob/d0471371a27f83beed5873fd4a1321ad76824877/python/readers/image_words_reader.py#L70 to

vertex_text = tf.reshape(vertex_text, shape=(-1, self.num_max_vertices, self.max_word_length)) ## Added -1 for shape attribute.

Sharathmk99 commented 4 years ago

@kmanojkkmr Were you able to train the model? If yes, were you able to run inference using the trained model?

kmanojkkmr commented 4 years ago

@Sharathmk99 Yes, I am able to train the model, but only if I change the value of "is_sampling_balanced" from 1 to 0 in config.ini file. I am only able to train for 19 iterations. :(

Sharathmk99 commented 4 years ago

@kmanojkkmr ok cool. I was finally able to train model for 5000 iterations and manually stoped the training, because accuracy was good.

Now I'm figuring out inference part.

Thank you for your response.

kmanojkkmr commented 4 years ago

@Sharathmk99 Great. Could you please share your config.ini file and what's the configurations of your system which you tried on? Like Is it CPU/GPU system? OS ? Tensorflow Version ?

Thanks in advance.