search

mesolitica / NLP-Models-Tensorflow

Gathers machine learning and Tensorflow deep learning models for NLP problems, 1.13 < Tensorflow < 2.0
MIT License
1.78k stars 729 forks source link

Spelling Correction- Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522]. #22

Open HarshithaMG opened 4 years ago

HarshithaMG commented 4 years ago

@huseinzol05 Can you please help me solve the below error??

Versions: Python: 3.6.10 Tensorflow: 1.13.1 Bert: 2.2.0

Code Source: https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/spelling-correction/3.bert-base-fast.ipynb

I am running the exact same code that is in the above code source link, but getting the below attached error while running the below chunk of code :

Code: tf.reset_default_graph() sess = tf.InteractiveSession() model = Model() sess.run(tf.global_variables_initializer()) var_lists = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope = 'bert') Error:

InvalidArgumentError Traceback (most recent call last) ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs) 1658 try: -> 1659 c_op = c_api.TF_FinishOperation(op_desc) 1660 except errors.InvalidArgumentError as e:

InvalidArgumentError: Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)

in 1 tf.reset_default_graph() 2 sess = tf.InteractiveSession() ----> 3 model = Model() 4 5 sess.run(tf.global_variables_initializer()) in __init__(self) 32 initializer = tf.zeros_initializer(), 33 ) ---> 34 logits = tf.matmul(input_tensor, tf.transpose(embedding)) 35 self.logits = tf.nn.bias_add(logits, output_bias) ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py in matmul(a, b, transpose_a, transpose_b, adjoint_a, adjoint_b, a_is_sparse, b_is_sparse, name) 2453 else: 2454 return gen_math_ops.mat_mul( -> 2455 a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) 2456 2457 ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py in mat_mul(a, b, transpose_a, transpose_b, name) 5331 _, _, _op = _op_def_lib._apply_op_helper( 5332 "MatMul", a=a, b=b, transpose_a=transpose_a, transpose_b=transpose_b, -> 5333 name=name) 5334 _result = _op.outputs[:] 5335 _inputs_flat = _op.inputs ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords) 786 op = g.create_op(op_type_name, inputs, output_types, name=scope, 787 input_types=input_types, attrs=attr_protos, --> 788 op_def=op_def) 789 return output_structure, op_def.is_stateful, op 790 ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs) 505 'in a future version' if date is None else ('after %s' % date), 506 instructions) --> 507 return func(*args, **kwargs) 508 509 doc = _add_deprecated_arg_notice_to_docstring( ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in create_op(***failed resolving arguments***) 3298 input_types=input_types, 3299 original_op=self._default_original_op, -> 3300 op_def=op_def) 3301 self._create_op_helper(ret, compute_device=compute_device) 3302 return ret ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in __init__(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def) 1821 op_def, inputs, node_def.attr) 1822 self._c_op = _create_c_op(self._graph, node_def, grouped_inputs, -> 1823 control_input_ops) 1824 1825 # Initialize self._outputs. ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs) 1660 except errors.InvalidArgumentError as e: 1661 # Convert to ValueError for backwards compatibility. -> 1662 raise ValueError(str(e)) 1663 1664 return c_op ValueError: Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522]. ---------------------------------------------------------------------------------------------------
huseinzol05 commented 4 years ago

I assume you downloaded wrong bert model.

XiaoxueGu commented 4 years ago

@huseinzol05 Can you please help me solve the below error??

Versions: Python: 3.6.10 Tensorflow: 1.13.1 Bert: 2.2.0

Code Source: https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/spelling-correction/3.bert-base-fast.ipynb

I am running the exact same code that is in the above code source link, but getting the below attached error while running the below chunk of code :

Code:

tf.reset_default_graph() sess = tf.InteractiveSession() model = Model() sess.run(tf.global_variables_initializer()) var_lists = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope = 'bert') Error: InvalidArgumentError Traceback (most recent call last) ~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs) 1658 try: -> 1659 c_op = c_api.TF_FinishOperation(op_desc) 1660 except errors.InvalidArgumentError as e:

InvalidArgumentError: Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) in 1 tf.reset_default_graph() 2 sess = tf.InteractiveSession() ----> 3 model = Model() 4 5 sess.run(tf.global_variables_initializer())

in init(self) 32 initializer = tf.zeros_initializer(), 33 ) ---> 34 logits = tf.matmul(input_tensor, tf.transpose(embedding)) 35 self.logits = tf.nn.bias_add(logits, output_bias)

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py in matmul(a, b, transpose_a, transpose_b, adjoint_a, adjoint_b, a_is_sparse, b_is_sparse, name) 2453 else: 2454 return gen_math_ops.mat_mul( -> 2455 a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) 2456 2457

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py in mat_mul(a, b, transpose_a, transposeb, name) 5331 , _, _op = _op_def_lib._apply_op_helper( 5332 "MatMul", a=a, b=b, transpose_a=transpose_a, transpose_b=transpose_b, -> 5333 name=name) 5334 _result = _op.outputs[:] 5335 _inputs_flat = _op.inputs

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords) 786 op = g.create_op(op_type_name, inputs, output_types, name=scope, 787 input_types=input_types, attrs=attr_protos, --> 788 op_def=op_def) 789 return output_structure, op_def.is_stateful, op 790

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, *kwargs) 505 'in a future version' if date is None else ('after %s' % date), 506 instructions) --> 507 return func(args, **kwargs) 508 509 doc = _add_deprecated_arg_notice_to_docstring(

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in createop(failed resolving arguments_) 3298 input_types=input_types, 3299 original_op=self._default_original_op, -> 3300 op_def=op_def) 3301 self._create_op_helper(ret, compute_device=compute_device) 3302 return ret

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in init(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def) 1821 op_def, inputs, node_def.attr) 1822 self._c_op = _create_c_op(self._graph, node_def, grouped_inputs, -> 1823 control_input_ops) 1824 1825 # Initialize self._outputs.

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs) 1660 except errors.InvalidArgumentError as e: 1661 # Convert to ValueError for backwards compatibility. -> 1662 raise ValueError(str(e)) 1663 1664 return c_op

ValueError: Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

By changing the shape, you can solve the problem. Adding the follow line on author's code output_layer = tf.reshape(output_layer, [-1, hidden_size]) preds = np.reshape(preds, [batch_size, seq_len, vocab_size])

deepang17 commented 4 years ago

By changing the shape, you can solve the problem. Adding the follow line on author's code output_layer = tf.reshape(output_layer, [-1, hidden_size]) preds = np.reshape(preds, [batch_size, seq_len, vocab_size])

This solution works.

To be more specific to the solution one can do the following:

  1. In the class Model, specify the line at:

    model = modeling.BertModel( config=bert_config, is_training=False, input_ids=self.X, use_one_hot_embeddings=False) output_layer = model.get_sequence_output() embedding = model.get_embedding_table() output_layer = tf.reshape(output_layer, [-1, bert_config.hidden_size])

  2. For the second part:

    preds = sess.run(tf.nn.log_softmax(model.logits), feed_dict = {model.X: masked_padded}) preds = np.reshape(preds, [masked_padded.shape[0], masked_padded.shape[1], 30522]) preds.shape

@huseinzol05 Thank you for repository. @HarshithaMG Thank you for the question. @XiaoxueGu Thank you for the solution.