Can't parse serialized Example

[robinlin1973]

Looks like failed to load the model from the ckpt, any hint? thanks.

I0731 15:33:06.451731 140067755902720 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /home/ec2-user/robin/clinicalBERT/output/model/model.ckpt. 2022-07-31 15:33:31.584083: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-07-31 15:33:31.795626: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_ids. Can't parse serialized Example. 2022-07-31 15:33:31.795633: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_mask. Can't parse serialized Example. 2022-07-31 15:33:31.795777: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_ids. Can't parse serialized Example. 2022-07-31 15:33:31.795937: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_ids. Can't parse serialized Example. 2022-07-31 15:33:31.795937: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_mask. Can't parse serialized Example. 2022-07-31 15:33:31.796025: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_ids. Can't parse serialized Example. 2022-07-31 15:33:31.796329: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example. 2022-07-31 15:33:31.796348: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_ids. Can't parse serialized Example. 2022-07-31 15:33:31.796660: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_mask. Can't parse serialized Example. 2022-07-31 15:33:31.796796: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: segment_ids. Can't parse serialized Example. 2022-07-31 15:33:31.796889: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_mask. Can't parse serialized Example. 2022-07-31 15:33:31.796975: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_weights. Can't parse serialized Example. 2022-07-31 15:33:31.797047: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example. 2022-07-31 15:33:31.797134: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_ids. Can't parse serialized Example. 2022-07-31 15:33:31.797213: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example. 2022-07-31 15:33:31.797291: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: input_ids. Can't parse serialized Example. ERROR:tensorflow:Error recorded from training_loop: 2 root error(s) found. (0) Invalid argument: Key: input_ids. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[IteratorGetNext]] (1) Invalid argument: Key: input_ids. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[IteratorGetNext]] [[IteratorGetNext/_4973]]

[robinlin1973]

Turns out the parameter in [finetune_lm_tf.sh] [create_pretrain_data.sh] have to be matched.