do_eval run error, fp16 model

[cronousbaby]

Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from ner_output/fp16_model.ckpt INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. 2020-02-15 11:34:17.949354: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2020-02-15 11:34:18.118593: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118599: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118644: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118689: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118665: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118725: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118602: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118602: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118644: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118752: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118779: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118599: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118813: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118747: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118619: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118705: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118599: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.118914: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.119047: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.119077: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-15 11:34:18.119123: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. INFO:tensorflow:Error recorded from evaluation_loop: Key: label_ids. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[node IteratorGetNext (defined at /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/util.py:110) ]] [[node confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert/data_2 (defined at /home/test/FasterTransformer/FasterTransformer_Bert/bert/tf_metrics.py:45) ]] INFO:tensorflow:evaluation_loop marked as finished WARNING:tensorflow:Reraising captured error

[swethmandava]

Can you link the whole log please?

[cronousbaby]

thanks! i linked the whole log. /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /home/test/.local/lib/python3.6/site-packages/absl/flags/_validators.py:359: UserWarning: Flag --data_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) /home/test/.local/lib/python3.6/site-packages/absl/flags/_validators.py:359: UserWarning: Flag --task_name has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) /home/test/.local/lib/python3.6/site-packages/absl/flags/_validators.py:359: UserWarning: Flag --vocab_file has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) /home/test/.local/lib/python3.6/site-packages/absl/flags/_validators.py:359: UserWarning: Flag --bert_config_file has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) /home/test/.local/lib/python3.6/site-packages/absl/flags/_validators.py:359: UserWarning: Flag --output_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name)

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder..model_fn at 0x7fe1e7485488>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Using config: {'_model_dir': 'ner_output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe1e0311208>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None} INFO:tensorflow:_TPUContext: eval_on_tpu True WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False. INFO:tensorflow:Writing example 0 of 68 INFO:tensorflow: Example INFO:tensorflow:guid: test-0 INFO:tensorflow:tokens: 美 国 的 华 莱 士 ， 我 和 他 谈 笑 风 生 。 INFO:tensorflow:input_ids: 101 5401 1744 4638 1290 5812 1894 8024 2769 1469 800 6448 5010 7599 4495 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:label_ids: 9 6 7 1 2 2 2 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow: Example INFO:tensorflow:guid: test-1 INFO:tensorflow:tokens: 看 包 公 断 案 的 戏 ， 看 他 威 风 凛 凛 坐 公 堂 拍 桌 子 动 刑 具 ， 多 少 还 有 一 点 担 心 ， 总 怕 靠 这 一 套 办 法 弄 出 错 案 来 ， 放 过 了 真 正 的 坏 人 ； INFO:tensorflow:input_ids: 101 4692 1259 1062 3171 3428 4638 2767 8024 4692 800 2014 7599 1123 1123 1777 1062 1828 2864 3430 2094 1220 1152 1072 8024 1914 2208 6820 3300 671 4157 2857 2552 8024 2600 2586 7479 6821 671 1947 1215 3791 2462 1139 7231 3428 3341 8024 3123 6814 749 4696 3633 4638 1776 782 8039 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:label_ids: 9 1 2 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow: Example INFO:tensorflow:guid: test-2 INFO:tensorflow:tokens: 可 看 《 包 公 赶 驴 》 这 出 戏 ， 心 里 就 很 踏 实 ： 这 样 是 一 断 一 个 准 的 。 INFO:tensorflow:input_ids: 101 1377 4692 517 1259 1062 6628 7723 518 6821 1139 2767 8024 2552 7027 2218 2523 6672 2141 8038 6821 3416 3221 671 3171 671 702 1114 4638 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:label_ids: 9 1 1 1 2 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow: Example INFO:tensorflow:guid: test-3 INFO:tensorflow:tokens: 譬 如 看 《 施 公 案 》 ， 施 大 人 坐 公 堂 问 案 子 不 得 要 领 ， 总 是 扮 成 普 通 百 姓 深 入 民 间 暗 中 查 访 ， 结 果 就 屡 破 奇 案 了 。 INFO:tensorflow:input_ids: 101 6357 1963 4692 517 3177 1062 3428 518 8024 3177 1920 782 1777 1062 1828 7309 3428 2094 679 2533 6206 7566 8024 2600 3221 2815 2768 3249 6858 4636 1998 3918 1057 3696 7313 3266 704 3389 6393 8024 5310 3362 2218 2249 4788 1936 3428 749 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:label_ids: 9 1 1 1 1 2 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow: Example INFO:tensorflow:guid: test-4 INFO:tensorflow:tokens: 如 果 有 人 问 我 ： [UNK] 你 看 过 许 多 包 公 戏 ， 哪 一 出 最 好 ？ [UNK] INFO:tensorflow:input_ids: 101 1963 3362 3300 782 7309 2769 8038 100 872 4692 6814 6387 1914 1259 1062 2767 8024 1525 671 1139 3297 1962 8043 100 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow:label_ids: 9 1 1 1 1 1 1 1 1 1 1 1 1 1 2 3 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 INFO:tensorflow: Running prediction INFO:tensorflow: Num examples = 68 (68 actual, 0 padding) INFO:tensorflow: Batch size = 8 WARNING:tensorflow:From /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /home/test/FasterTransformer/FasterTransformer_Bert/fast_infer_util.py:76: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.experimental.map_and_batch(...). WARNING:tensorflow:From /home/test/FasterTransformer/FasterTransformer_Bert/fast_infer_util.py:55: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Running infer on CPU INFO:tensorflow: Features INFO:tensorflow: name = input_ids, shape = (8, 128) INFO:tensorflow: name = input_mask, shape = (8, 128) INFO:tensorflow: name = is_real_example, shape = (8,) INFO:tensorflow: name = label_ids, shape = (8,) INFO:tensorflow: name = segment_ids, shape = (8, 128) WARNING:tensorflow:From /home/test/FasterTransformer/FasterTransformer_Bert/my_modeling.py:676: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. INFO:tensorflow: Trainable Variables INFO:tensorflow: name = bert/embeddings/word_embeddings:0, shape = (21128, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/position_embeddings:0, shape = (512, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = output_weights:0, shape = (11, 768), INIT_FROM_CKPT INFO:tensorflow: name = output_bias:0, shape = (11,), INIT_FROM_CKPT INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Graph was finalized. 2020-02-20 08:16:28.889493: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2020-02-20 08:16:29.070817: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x56325bcd77a0 executing computations on platform CUDA. Devices: 2020-02-20 08:16:29.070871: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5 2020-02-20 08:16:29.070886: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): GeForce RTX 2080 Ti, Compute Capability 7.5 2020-02-20 08:16:29.097831: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3500000000 Hz 2020-02-20 08:16:29.098898: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x56325bccb150 executing computations on platform Host. Devices: 2020-02-20 08:16:29.098920: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , 2020-02-20 08:16:29.099081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:17:00.0 totalMemory: 10.76GiB freeMemory: 10.45GiB 2020-02-20 08:16:29.099120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:65:00.0 totalMemory: 10.76GiB freeMemory: 816.00MiB 2020-02-20 08:16:29.099183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1 2020-02-20 08:16:29.101090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-02-20 08:16:29.101111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1 2020-02-20 08:16:29.101120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N N 2020-02-20 08:16:29.101126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N N 2020-02-20 08:16:29.101216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10166 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5) 2020-02-20 08:16:29.101649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 591 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from ner_output/fp16_model.ckpt INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. 2020-02-20 08:16:30.245222: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2020-02-20 08:16:30.420409: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420475: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420512: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420477: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420475: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420588: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420619: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420661: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420639: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420558: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420703: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420796: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420837: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420887: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420901: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420960: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420962: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. 2020-02-20 08:16:30.420970: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: label_ids. Can't parse serialized Example. INFO:tensorflow:Error recorded from prediction_loop: Key: label_ids. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[node IteratorGetNext (defined at /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/util.py:110) ]] [[node IteratorGetNext (defined at /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/util.py:110) ]] INFO:tensorflow:prediction_loop marked as finished WARNING:tensorflow:Reraising captured error Traceback (most recent call last): File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Key: label_ids. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[{{node IteratorGetNext}}]] [[{{node IteratorGetNext}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "run_ner_fastertf.py", line 76, in tf.app.run() File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/home/test/FasterTransformer/FasterTransformer_Bert/bert/run_ner.py", line 973, in main for prediction in result: File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2500, in predict rendezvous.raise_errors() File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 128, in raise_errors six.reraise(typ, value, traceback) File "/home/test/.local/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2494, in predict yield_single_examples=yield_single_examples): File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 629, in predict preds_evaluated = mon_sess.run(predictions) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run run_metadata=run_metadata) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run run_metadata=run_metadata) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run raise six.reraise(original_exc_info) File "/home/test/.local/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run return self._sess.run(args, *kwargs) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1327, in run run_metadata=run_metadata) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1091, in run return self._sess.run(args, **kwargs) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Key: label_ids. Can't parse serialized Example. [[{{node ParseSingleExample/ParseSingleExample}}]] [[node IteratorGetNext (defined at /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/util.py:110) ]] [[node IteratorGetNext (defined at /home/test/anaconda3/envs/fastertransformer/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/util.py:110) ]]

[swethmandava]

Please specify the following to help us debug:

1. Launch Command
2. What data are you using
3. Did you make any modifications to the code?
4. Are you using a checkpoint?

Thank you!

[cronousbaby]

My Test Is a Named Entity Recognition Program base on bert

[swethmandava]

Closing due to inactivity!