search

google-research / bert

TensorFlow code and pre-trained models for BERT
https://arxiv.org/abs/1810.04805
Apache License 2.0
37.47k stars 9.54k forks source link

tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example. #1365

Open ghost opened 1 year ago

ghost commented 1 year ago

When I run the run_pretraining.py file, the code runs well, but this error occurs. I don't know what the problem is. I set max_predictions_per_seq=20 and ran. In the create_pretraining_data.py file, max_predictions_per_seq is also 20. set max_seq_length=128

NFO:tensorflow:global_step/sec: 2.5727 I0830 19:28:21.643072 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.5727 INFO:tensorflow:examples/sec: 82.3263 I0830 19:28:21.644017 140575351207744 tpu_estimator.py:2308] examples/sec: 82.3263 2022-08-30 19:28:21.647980: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example. INFO:tensorflow:global_step/sec: 2.58677 I0830 19:28:22.029351 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.58677 INFO:tensorflow:examples/sec: 82.7765 I0830 19:28:22.029989 140575351207744 tpu_estimator.py:2308] examples/sec: 82.7765 INFO:tensorflow:global_step/sec: 2.57967 I0830 19:28:22.416983 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.57967 INFO:tensorflow:examples/sec: 82.5493 I0830 19:28:22.417581 140575351207744 tpu_estimator.py:2308] examples/sec: 82.5493 INFO:tensorflow:global_step/sec: 2.57398 I0830 19:28:22.805490 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.57398 INFO:tensorflow:examples/sec: 82.3672 I0830 19:28:22.806136 140575351207744 tpu_estimator.py:2308] examples/sec: 82.3672 INFO:tensorflow:global_step/sec: 2.59985 I0830 19:28:23.189946 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.59985 INFO:tensorflow:examples/sec: 83.1952 I0830 19:28:23.190213 140575351207744 tpu_estimator.py:2308] examples/sec: 83.1952 ERROR:tensorflow:Error recorded from training_loop: {{function_node inference_tf_data_experimental_map_andbatch_83}} Key: masked_lm_positions. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[IteratorGetNext]] E0830 19:28:23.417265 140575351207744 error_handling.py:75] Error recorded from training_loop: {{function_node __inference_tf_data_experimental_map_andbatch_83}} Key: masked_lm_positions. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[IteratorGetNext]] INFO:tensorflow:training_loop marked as finished I0830 19:28:23.417502 140575351207744 error_handling.py:101] training_loop marked as finished WARNING:tensorflow:Reraising captured error W0830 19:28:23.417612 140575351207744 error_handling.py:135] Reraising captured error Traceback (most recent call last): File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node inference_tf_data_experimental_map_andbatch_83}} Key: masked_lm_positions. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[IteratorGetNext]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "run_pretraining.py", line 493, in tf.app.run() File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/absl/app.py", line 312, in run _run_main(main, args) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "run_pretraining.py", line 466, in main estimator.train(input_fn=train_input_fn, max_steps=FLAGS.num_train_steps) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3035, in train rendezvous.raise_errors() File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 136, in raise_errors six.reraise(typ, value, traceback) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/six.py", line 719, in reraise raise value File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3030, in train saving_listeners=saving_listeners) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimatorspec , loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(original_exc_info) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/six.py", line 719, in reraise raise value File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(args, *kwargs) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run run_metadata=run_metadata) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run return self._sess.run(args, **kwargs) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __inference_tf_data_experimental_map_andbatch_83}} Key: masked_lm_positions. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[IteratorGetNext]]

pzhang84 commented 1 year ago

I got the same issue. Did you solve that?

minotskaya commented 1 year ago

Same here... Any ideas how to solve?