The prediction works great and the results are written to my output folder. But for some reason, right afterwards the script tries to add the model to an "eval list" and gets stuck in a loop doing this. This can be seen in the following log, Id really appreciate any ideas on this issue:
####################
(dynamic_QA) > python sandbox.py
2021-03-29 13:26:32.050191: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2021-03-29 13:26:32.050366: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
INFO:tensorflow:loading sentence piece model
I0329 13:26:34.127707 10180 tokenization.py:188] loading sentence piece model
WARNING:tensorflow:Estimator's model_fn (<function v2_model_fn_builder..model_fn at 0x000001AB18F8A168>) includes params argument, but params are not passed to Estimator.
W0329 13:26:35.338461 10180 estimator.py:1994] Estimator's model_fn (<function v2_model_fn_builder..model_fn at 0x000001AB18F8A168>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using config: {'_model_dir': 'albert/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001AB2198B708>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I0329 13:26:35.340723 10180 estimator.py:212] Using config: {'_model_dir': 'albert/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001AB2198B708>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu True
I0329 13:26:35.342096 10180 tpu_context.py:220] _TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0329 13:26:35.342610 10180 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False.
INFO:tensorflow:Loading eval features from albert/feature_files/predict_feature_left_file2.tfrecord
I0329 13:26:35.343953 10180 run_squad_v2.py:338] Loading eval features from albert/feature_files/predict_feature_left_file2.tfrecord
INFO:tensorflow: Running predictions
I0329 13:26:35.405187 10180 run_squad_v2.py:364] Running predictions
INFO:tensorflow: Num orig examples = 8
I0329 13:26:35.407440 10180 run_squad_v2.py:365] Num orig examples = 8
INFO:tensorflow: Num split examples = 8
I0329 13:26:35.407887 10180 run_squad_v2.py:366] Num split examples = 8
INFO:tensorflow: Batch size = 8
I0329 13:26:35.408218 10180 run_squad_v2.py:367] Batch size = 8
WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass _constraint arguments to layers.
W0329 13:26:35.417991 10180 deprecation.py:506] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass _constraint arguments to layers.
WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:703: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.map_and_batch(...).
W0329 13:26:35.431954 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:703: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.map_and_batch(...).
WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\contrib\data\python\ops\batching.py:276: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map(map_func, num_parallel_calls) followed by tf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation.
W0329 13:26:35.433997 10180 deprecation.py:323] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\contrib\data\python\ops\batching.py:276: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map(map_func, num_parallel_calls) followed by tf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation.
WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:680: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0329 13:26:35.578533 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:680: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
INFO:tensorflow:Calling model_fn.
I0329 13:26:35.591268 10180 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Running infer on CPU
I0329 13:26:35.592265 10180 tpu_estimator.py:3124] Running infer on CPU
INFO:tensorflow: Features
I0329 13:26:35.592265 10180 squad_utils.py:1589] Features
INFO:tensorflow: name = input_ids, shape = (?, 384)
I0329 13:26:35.593265 10180 squad_utils.py:1591] name = input_ids, shape = (?, 384)
INFO:tensorflow: name = input_mask, shape = (?, 384)
I0329 13:26:35.593265 10180 squad_utils.py:1591] name = input_mask, shape = (?, 384)
INFO:tensorflow: name = p_mask, shape = (?, 384)
I0329 13:26:35.593265 10180 squad_utils.py:1591] name = p_mask, shape = (?, 384)
INFO:tensorflow: name = segment_ids, shape = (?, 384)
I0329 13:26:35.594565 10180 squad_utils.py:1591] name = segment_ids, shape = (?, 384)
INFO:tensorflow: name = unique_ids, shape = (?,)
I0329 13:26:35.594916 10180 squad_utils.py:1591] name = unique_ids, shape = (?,)
INFO:tensorflow:creating model from albert_config
I0329 13:26:35.595249 10180 fine_tuning_utils.py:69] creating model from albert_config
WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\modeling.py:256: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
W0329 13:26:36.803697 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\modeling.py:256: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\layers\core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0329 13:26:36.805737 10180 deprecation.py:323] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\layers\core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:1565: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
W0329 13:26:36.950624 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:1565: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
INFO:tensorflow:name bert/embeddings/word_embeddings match to bert/embeddings/word_embeddings
I0329 13:26:36.964367 10180 modeling.py:392] name bert/embeddings/word_embeddings match to bert/embeddings/word_embeddings
INFO:tensorflow:name bert/embeddings/token_type_embeddings match to bert/embeddings/token_type_embeddings
I0329 13:26:36.965355 10180 modeling.py:392] name bert/embeddings/token_type_embeddings match to bert/embeddings/token_type_embeddings
INFO:tensorflow:name bert/embeddings/position_embeddings match to bert/embeddings/position_embeddings
I0329 13:26:36.965355 10180 modeling.py:392] name bert/embeddings/position_embeddings match to bert/embeddings/position_embeddings
INFO:tensorflow:name bert/embeddings/LayerNorm/beta match to bert/embeddings/LayerNorm/beta
I0329 13:26:36.965355 10180 modeling.py:392] name bert/embeddings/LayerNorm/beta match to bert/embeddings/LayerNorm/beta
INFO:tensorflow:name bert/embeddings/LayerNorm/gamma match to bert/embeddings/LayerNorm/gamma
I0329 13:26:36.966348 10180 modeling.py:392] name bert/embeddings/LayerNorm/gamma match to bert/embeddings/LayerNorm/gamma
INFO:tensorflow:name bert/encoder/embedding_hidden_mapping_in/kernel match to bert/encoder/embedding_hidden_mapping_in/kernel
I0329 13:26:36.966348 10180 modeling.py:392] name bert/encoder/embedding_hidden_mapping_in/kernel match to bert/encoder/embedding_hidden_mapping_in/kernel
INFO:tensorflow:name bert/encoder/embedding_hidden_mapping_in/bias match to bert/encoder/embedding_hidden_mapping_in/bias
I0329 13:26:36.966348 10180 modeling.py:392] name bert/encoder/embedding_hidden_mapping_in/bias match to bert/encoder/embedding_hidden_mapping_in/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel
I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias
I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel
I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias
I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel
I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias
I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel
I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias
I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta
I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma
I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel
I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias
I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel
I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias
I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta
I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta
INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma
I0329 13:26:36.970337 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma
INFO:tensorflow:name bert/pooler/dense/kernel match to bert/pooler/dense/kernel
I0329 13:26:36.970337 10180 modeling.py:392] name bert/pooler/dense/kernel match to bert/pooler/dense/kernel
INFO:tensorflow:name bert/pooler/dense/bias match to bert/pooler/dense/bias
I0329 13:26:36.970337 10180 modeling.py:392] name bert/pooler/dense/bias match to bert/pooler/dense/bias
INFO:tensorflow:name start_logits/dense/kernel match to start_logits/dense/kernel
I0329 13:26:36.970337 10180 modeling.py:392] name start_logits/dense/kernel match to start_logits/dense/kernel
INFO:tensorflow:name start_logits/dense/bias match to start_logits/dense/bias
I0329 13:26:36.977330 10180 modeling.py:392] name start_logits/dense/bias match to start_logits/dense/bias
INFO:tensorflow:name end_logits/dense_0/kernel match to end_logits/dense_0/kernel
I0329 13:26:36.977908 10180 modeling.py:392] name end_logits/dense_0/kernel match to end_logits/dense_0/kernel
INFO:tensorflow:name end_logits/dense_0/bias match to end_logits/dense_0/bias
I0329 13:26:36.978228 10180 modeling.py:392] name end_logits/dense_0/bias match to end_logits/dense_0/bias
INFO:tensorflow:name end_logits/LayerNorm/beta match to end_logits/LayerNorm/beta
I0329 13:26:36.978543 10180 modeling.py:392] name end_logits/LayerNorm/beta match to end_logits/LayerNorm/beta
INFO:tensorflow:name end_logits/LayerNorm/gamma match to end_logits/LayerNorm/gamma
I0329 13:26:36.978868 10180 modeling.py:392] name end_logits/LayerNorm/gamma match to end_logits/LayerNorm/gamma
INFO:tensorflow:name end_logits/dense_1/kernel match to end_logits/dense_1/kernel
I0329 13:26:36.979185 10180 modeling.py:392] name end_logits/dense_1/kernel match to end_logits/dense_1/kernel
INFO:tensorflow:name end_logits/dense_1/bias match to end_logits/dense_1/bias
I0329 13:26:36.979353 10180 modeling.py:392] name end_logits/dense_1/bias match to end_logits/dense_1/bias
INFO:tensorflow:name answer_class/dense_0/kernel match to answer_class/dense_0/kernel
I0329 13:26:36.979353 10180 modeling.py:392] name answer_class/dense_0/kernel match to answer_class/dense_0/kernel
INFO:tensorflow:name answer_class/dense_0/bias match to answer_class/dense_0/bias
I0329 13:26:36.979353 10180 modeling.py:392] name answer_class/dense_0/bias match to answer_class/dense_0/bias
INFO:tensorflow:name answer_class/dense_1/kernel match to answer_class/dense_1/kernel
I0329 13:26:36.980341 10180 modeling.py:392] name answer_class/dense_1/kernel match to answer_class/dense_1/kernel
INFO:tensorflow: Trainable Variables
I0329 13:26:37.055370 10180 squad_utils.py:1631] Trainable Variables
INFO:tensorflow: name = bert/embeddings/word_embeddings:0, shape = (30000, 128), INIT_FROM_CKPT
I0329 13:26:37.056367 10180 squad_utils.py:1637] name = bert/embeddings/word_embeddings:0, shape = (30000, 128), INIT_FROM_CKPT
INFO:tensorflow: name = bert/embeddings/token_type_embeddings:0, shape = (2, 128), INIT_FROM_CKPT
I0329 13:26:37.057364 10180 squad_utils.py:1637] name = bert/embeddings/token_type_embeddings:0, shape = (2, 128), INIT_FROM_CKPT
INFO:tensorflow: name = bert/embeddings/position_embeddings:0, shape = (512, 128), INIT_FROM_CKPT
I0329 13:26:37.057364 10180 squad_utils.py:1637] name = bert/embeddings/position_embeddings:0, shape = (512, 128), INIT_FROM_CKPT
INFO:tensorflow: name = bert/embeddings/LayerNorm/beta:0, shape = (128,), INIT_FROM_CKPT
I0329 13:26:37.057364 10180 squad_utils.py:1637] name = bert/embeddings/LayerNorm/beta:0, shape = (128,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/embeddings/LayerNorm/gamma:0, shape = (128,), INIT_FROM_CKPT
I0329 13:26:37.058362 10180 squad_utils.py:1637] name = bert/embeddings/LayerNorm/gamma:0, shape = (128,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/embedding_hidden_mapping_in/kernel:0, shape = (128, 768), INIT_FROM_CKPT
I0329 13:26:37.058548 10180 squad_utils.py:1637] name = bert/encoder/embedding_hidden_mapping_in/kernel:0, shape = (128, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/embedding_hidden_mapping_in/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.058548 10180 squad_utils.py:1637] name = bert/encoder/embedding_hidden_mapping_in/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT
I0329 13:26:37.058548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT
I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT
I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT
I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT
I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT
I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT
I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT
I0329 13:26:37.062541 10180 squad_utils.py:1637] name = bert/pooler/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT
INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.062541 10180 squad_utils.py:1637] name = bert/pooler/dense/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = start_logits/dense/kernel:0, shape = (768, 1), INIT_FROM_CKPT
I0329 13:26:37.062541 10180 squad_utils.py:1637] name = start_logits/dense/kernel:0, shape = (768, 1), INIT_FROM_CKPT
INFO:tensorflow: name = start_logits/dense/bias:0, shape = (1,), INIT_FROM_CKPT
I0329 13:26:37.062541 10180 squad_utils.py:1637] name = start_logits/dense/bias:0, shape = (1,), INIT_FROM_CKPT
INFO:tensorflow: name = end_logits/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT
I0329 13:26:37.062541 10180 squad_utils.py:1637] name = end_logits/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT
INFO:tensorflow: name = end_logits/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.062541 10180 squad_utils.py:1637] name = end_logits/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = end_logits/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = end_logits/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = end_logits/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT
I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT
INFO:tensorflow: name = end_logits/dense_1/bias:0, shape = (1,), INIT_FROM_CKPT
I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/dense_1/bias:0, shape = (1,), INIT_FROM_CKPT
INFO:tensorflow: name = answer_class/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT
I0329 13:26:37.063537 10180 squad_utils.py:1637] name = answer_class/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT
INFO:tensorflow: name = answer_class/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT
I0329 13:26:37.063537 10180 squad_utils.py:1637] name = answer_class/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT
INFO:tensorflow: name = answer_class/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT
I0329 13:26:37.064536 10180 squad_utils.py:1637] name = answer_class/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT
INFO:tensorflow:Done calling model_fn.
I0329 13:26:37.064536 10180 estimator.py:1150] Done calling model_fn.
WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0329 13:26:37.093492 10180 deprecation.py:323] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
INFO:tensorflow:Graph was finalized.
I0329 13:26:37.152336 10180 monitored_session.py:240] Graph was finalized.
2021-03-29 13:26:37.154873: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-03-29 13:26:37.158136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-03-29 13:26:37.168417: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2021-03-29 13:26:37.172695: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-GV232QR
2021-03-29 13:26:37.172943: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-GV232QR
INFO:tensorflow:Restoring parameters from albert/output\model.ckpt-best
I0329 13:26:37.177373 10180 saver.py:1284] Restoring parameters from albert/output\model.ckpt-best
INFO:tensorflow:Running local_init_op.
I0329 13:26:37.463492 10180 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0329 13:26:37.531100 10180 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Processing example: 0
I0329 13:26:42.834813 10180 run_squad_v2.py:389] Processing example: 0
INFO:tensorflow:prediction_loop marked as finished
I0329 13:26:42.883660 10180 error_handling.py:101] prediction_loop marked as finished
INFO:tensorflow:prediction_loop marked as finished
I0329 13:26:42.884687 10180 error_handling.py:101] prediction_loop marked as finished
INFO:tensorflow:Writing predictions to: albert/output\predictions.json
I0329 13:26:42.886681 10180 squad_utils.py:1309] Writing predictions to: albert/output\predictions.json
INFO:tensorflow:Writing nbest to: albert/output\nbest_predictions.json
I0329 13:26:42.886681 10180 squad_utils.py:1310] Writing nbest to: albert/output\nbest_predictions.json
INFO:tensorflow:Writing predictions to: albert/output\predictions.json
I0329 13:26:42.908359 10180 squad_utils.py:1309] Writing predictions to: albert/output\predictions.json
INFO:tensorflow:Writing nbest to: albert/output\nbest_predictions.json
I0329 13:26:42.909356 10180 squad_utils.py:1310] Writing nbest to: albert/output\nbest_predictions.json
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.931493 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.932229 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.932597 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.933382 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.933740 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.933740 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.934739 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.934739 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.938729 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.938729 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.939726 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.939726 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.940616 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.941466 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.941834 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
I0329 13:26:42.942656 10180 run_squad_v2.py:464] found 1 files.
INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list.
I0329 13:26:42.943008 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list.
INFO:tensorflow:found 1 files.
pustar
commented
1 year ago
I am trying to predict answers for a new text with questions with a base albert model fine-tuned on squad2.0. These are my parameters:
python -m run_squad_v2 --albert_config_file albert/model/albert_config.json --train_file albert/squad/train-v2.0.json --predict_file albert/squad/qa_input.json --train_feature_file albert/feature_files/train_feature_file.tfrecord --predict_feature_file albert/feature_files/predict_feature_file2.tfrecord --predict_feature_left_file albert/feature_files/predict_feature_left_file2.tfrecord --output_dir albert/output --init_checkpoint albert/output/model.ckpt-8144 --spm_model_file albert/model/30k-clean.model --do_predict=True --do_train=False --do_eval=False --max_answer_length=30
The prediction works great and the results are written to my output folder. But for some reason, right afterwards the script tries to add the model to an "eval list" and gets stuck in a loop doing this. This can be seen in the following log, Id really appreciate any ideas on this issue:
####################
(dynamic_QA) > python sandbox.py 2021-03-29 13:26:32.050191: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found 2021-03-29 13:26:32.050366: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. INFO:tensorflow:loading sentence piece model I0329 13:26:34.127707 10180 tokenization.py:188] loading sentence piece model WARNING:tensorflow:Estimator's model_fn (<function v2_model_fn_builder..model_fn at 0x000001AB18F8A168>) includes params argument, but params are not passed to Estimator.
W0329 13:26:35.338461 10180 estimator.py:1994] Estimator's model_fn (<function v2_model_fn_builder..model_fn at 0x000001AB18F8A168>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using config: {'_model_dir': 'albert/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001AB2198B708>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I0329 13:26:35.340723 10180 estimator.py:212] Using config: {'_model_dir': 'albert/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 0, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001AB2198B708>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu True
I0329 13:26:35.342096 10180 tpu_context.py:220] _TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0329 13:26:35.342610 10180 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False.
INFO:tensorflow:Loading eval features from albert/feature_files/predict_feature_left_file2.tfrecord
I0329 13:26:35.343953 10180 run_squad_v2.py:338] Loading eval features from albert/feature_files/predict_feature_left_file2.tfrecord
INFO:tensorflow: Running predictions
I0329 13:26:35.405187 10180 run_squad_v2.py:364] Running predictions
INFO:tensorflow: Num orig examples = 8
I0329 13:26:35.407440 10180 run_squad_v2.py:365] Num orig examples = 8
INFO:tensorflow: Num split examples = 8
I0329 13:26:35.407887 10180 run_squad_v2.py:366] Num split examples = 8
INFO:tensorflow: Batch size = 8
I0329 13:26:35.408218 10180 run_squad_v2.py:367] Batch size = 8
WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass _constraint arguments to layers.
W0329 13:26:35.417991 10180 deprecation.py:506] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass _constraint arguments to layers.
WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:703: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use
tf.data.experimental.map_and_batch(...). W0329 13:26:35.431954 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:703: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Usetf.data.experimental.map_and_batch(...). WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\contrib\data\python\ops\batching.py:276: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Usetf.data.Dataset.map(map_func, num_parallel_calls)followed bytf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation. W0329 13:26:35.433997 10180 deprecation.py:323] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\contrib\data\python\ops\batching.py:276: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Usetf.data.Dataset.map(map_func, num_parallel_calls)followed bytf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation. WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:680: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Usetf.castinstead. W0329 13:26:35.578533 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:680: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Usetf.castinstead. INFO:tensorflow:Calling model_fn. I0329 13:26:35.591268 10180 estimator.py:1148] Calling model_fn. INFO:tensorflow:Running infer on CPU I0329 13:26:35.592265 10180 tpu_estimator.py:3124] Running infer on CPU INFO:tensorflow: Features I0329 13:26:35.592265 10180 squad_utils.py:1589] Features INFO:tensorflow: name = input_ids, shape = (?, 384) I0329 13:26:35.593265 10180 squad_utils.py:1591] name = input_ids, shape = (?, 384) INFO:tensorflow: name = input_mask, shape = (?, 384) I0329 13:26:35.593265 10180 squad_utils.py:1591] name = input_mask, shape = (?, 384) INFO:tensorflow: name = p_mask, shape = (?, 384) I0329 13:26:35.593265 10180 squad_utils.py:1591] name = p_mask, shape = (?, 384) INFO:tensorflow: name = segment_ids, shape = (?, 384) I0329 13:26:35.594565 10180 squad_utils.py:1591] name = segment_ids, shape = (?, 384) INFO:tensorflow: name = unique_ids, shape = (?,) I0329 13:26:35.594916 10180 squad_utils.py:1591] name = unique_ids, shape = (?,) INFO:tensorflow:creating model from albert_config I0329 13:26:35.595249 10180 fine_tuning_utils.py:69] creating model from albert_config WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\modeling.py:256: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.Dense instead. W0329 13:26:36.803697 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\modeling.py:256: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.Dense instead. WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\layers\core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please uselayer.__call__method instead. W0329 13:26:36.805737 10180 deprecation.py:323] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\layers\core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please uselayer.__call__method instead. WARNING:tensorflow:From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:1565: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dropout instead. W0329 13:26:36.950624 10180 deprecation.py:323] From C:\Users\alexa\PycharmProjects\bachelorarbeit\scripts\albert\squad_utils.py:1565: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dropout instead. INFO:tensorflow:name bert/embeddings/word_embeddings match to bert/embeddings/word_embeddings I0329 13:26:36.964367 10180 modeling.py:392] name bert/embeddings/word_embeddings match to bert/embeddings/word_embeddings INFO:tensorflow:name bert/embeddings/token_type_embeddings match to bert/embeddings/token_type_embeddings I0329 13:26:36.965355 10180 modeling.py:392] name bert/embeddings/token_type_embeddings match to bert/embeddings/token_type_embeddings INFO:tensorflow:name bert/embeddings/position_embeddings match to bert/embeddings/position_embeddings I0329 13:26:36.965355 10180 modeling.py:392] name bert/embeddings/position_embeddings match to bert/embeddings/position_embeddings INFO:tensorflow:name bert/embeddings/LayerNorm/beta match to bert/embeddings/LayerNorm/beta I0329 13:26:36.965355 10180 modeling.py:392] name bert/embeddings/LayerNorm/beta match to bert/embeddings/LayerNorm/beta INFO:tensorflow:name bert/embeddings/LayerNorm/gamma match to bert/embeddings/LayerNorm/gamma I0329 13:26:36.966348 10180 modeling.py:392] name bert/embeddings/LayerNorm/gamma match to bert/embeddings/LayerNorm/gamma INFO:tensorflow:name bert/encoder/embedding_hidden_mapping_in/kernel match to bert/encoder/embedding_hidden_mapping_in/kernel I0329 13:26:36.966348 10180 modeling.py:392] name bert/encoder/embedding_hidden_mapping_in/kernel match to bert/encoder/embedding_hidden_mapping_in/kernel INFO:tensorflow:name bert/encoder/embedding_hidden_mapping_in/bias match to bert/encoder/embedding_hidden_mapping_in/bias I0329 13:26:36.966348 10180 modeling.py:392] name bert/encoder/embedding_hidden_mapping_in/bias match to bert/encoder/embedding_hidden_mapping_in/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias I0329 13:26:36.967349 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma I0329 13:26:36.968342 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias match to bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta I0329 13:26:36.969339 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta INFO:tensorflow:name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma I0329 13:26:36.970337 10180 modeling.py:392] name bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma match to bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma INFO:tensorflow:name bert/pooler/dense/kernel match to bert/pooler/dense/kernel I0329 13:26:36.970337 10180 modeling.py:392] name bert/pooler/dense/kernel match to bert/pooler/dense/kernel INFO:tensorflow:name bert/pooler/dense/bias match to bert/pooler/dense/bias I0329 13:26:36.970337 10180 modeling.py:392] name bert/pooler/dense/bias match to bert/pooler/dense/bias INFO:tensorflow:name start_logits/dense/kernel match to start_logits/dense/kernel I0329 13:26:36.970337 10180 modeling.py:392] name start_logits/dense/kernel match to start_logits/dense/kernel INFO:tensorflow:name start_logits/dense/bias match to start_logits/dense/bias I0329 13:26:36.977330 10180 modeling.py:392] name start_logits/dense/bias match to start_logits/dense/bias INFO:tensorflow:name end_logits/dense_0/kernel match to end_logits/dense_0/kernel I0329 13:26:36.977908 10180 modeling.py:392] name end_logits/dense_0/kernel match to end_logits/dense_0/kernel INFO:tensorflow:name end_logits/dense_0/bias match to end_logits/dense_0/bias I0329 13:26:36.978228 10180 modeling.py:392] name end_logits/dense_0/bias match to end_logits/dense_0/bias INFO:tensorflow:name end_logits/LayerNorm/beta match to end_logits/LayerNorm/beta I0329 13:26:36.978543 10180 modeling.py:392] name end_logits/LayerNorm/beta match to end_logits/LayerNorm/beta INFO:tensorflow:name end_logits/LayerNorm/gamma match to end_logits/LayerNorm/gamma I0329 13:26:36.978868 10180 modeling.py:392] name end_logits/LayerNorm/gamma match to end_logits/LayerNorm/gamma INFO:tensorflow:name end_logits/dense_1/kernel match to end_logits/dense_1/kernel I0329 13:26:36.979185 10180 modeling.py:392] name end_logits/dense_1/kernel match to end_logits/dense_1/kernel INFO:tensorflow:name end_logits/dense_1/bias match to end_logits/dense_1/bias I0329 13:26:36.979353 10180 modeling.py:392] name end_logits/dense_1/bias match to end_logits/dense_1/bias INFO:tensorflow:name answer_class/dense_0/kernel match to answer_class/dense_0/kernel I0329 13:26:36.979353 10180 modeling.py:392] name answer_class/dense_0/kernel match to answer_class/dense_0/kernel INFO:tensorflow:name answer_class/dense_0/bias match to answer_class/dense_0/bias I0329 13:26:36.979353 10180 modeling.py:392] name answer_class/dense_0/bias match to answer_class/dense_0/bias INFO:tensorflow:name answer_class/dense_1/kernel match to answer_class/dense_1/kernel I0329 13:26:36.980341 10180 modeling.py:392] name answer_class/dense_1/kernel match to answer_class/dense_1/kernel INFO:tensorflow: Trainable Variables I0329 13:26:37.055370 10180 squad_utils.py:1631] Trainable Variables INFO:tensorflow: name = bert/embeddings/word_embeddings:0, shape = (30000, 128), INIT_FROM_CKPT I0329 13:26:37.056367 10180 squad_utils.py:1637] name = bert/embeddings/word_embeddings:0, shape = (30000, 128), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/token_type_embeddings:0, shape = (2, 128), INIT_FROM_CKPT I0329 13:26:37.057364 10180 squad_utils.py:1637] name = bert/embeddings/token_type_embeddings:0, shape = (2, 128), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/position_embeddings:0, shape = (512, 128), INIT_FROM_CKPT I0329 13:26:37.057364 10180 squad_utils.py:1637] name = bert/embeddings/position_embeddings:0, shape = (512, 128), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/LayerNorm/beta:0, shape = (128,), INIT_FROM_CKPT I0329 13:26:37.057364 10180 squad_utils.py:1637] name = bert/embeddings/LayerNorm/beta:0, shape = (128,), INIT_FROM_CKPT INFO:tensorflow: name = bert/embeddings/LayerNorm/gamma:0, shape = (128,), INIT_FROM_CKPT I0329 13:26:37.058362 10180 squad_utils.py:1637] name = bert/embeddings/LayerNorm/gamma:0, shape = (128,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/embedding_hidden_mapping_in/kernel:0, shape = (128, 768), INIT_FROM_CKPT I0329 13:26:37.058548 10180 squad_utils.py:1637] name = bert/encoder/embedding_hidden_mapping_in/kernel:0, shape = (128, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/embedding_hidden_mapping_in/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.058548 10180 squad_utils.py:1637] name = bert/encoder/embedding_hidden_mapping_in/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT I0329 13:26:37.058548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.059548 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/attention_1/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT I0329 13:26:37.060546 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.061542 10180 squad_utils.py:1637] name = bert/encoder/transformer/group_0/inner_group_0/LayerNorm_1/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT I0329 13:26:37.062541 10180 squad_utils.py:1637] name = bert/pooler/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.062541 10180 squad_utils.py:1637] name = bert/pooler/dense/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = start_logits/dense/kernel:0, shape = (768, 1), INIT_FROM_CKPT I0329 13:26:37.062541 10180 squad_utils.py:1637] name = start_logits/dense/kernel:0, shape = (768, 1), INIT_FROM_CKPT INFO:tensorflow: name = start_logits/dense/bias:0, shape = (1,), INIT_FROM_CKPT I0329 13:26:37.062541 10180 squad_utils.py:1637] name = start_logits/dense/bias:0, shape = (1,), INIT_FROM_CKPT INFO:tensorflow: name = end_logits/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT I0329 13:26:37.062541 10180 squad_utils.py:1637] name = end_logits/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT INFO:tensorflow: name = end_logits/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.062541 10180 squad_utils.py:1637] name = end_logits/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = end_logits/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = end_logits/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = end_logits/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT INFO:tensorflow: name = end_logits/dense_1/bias:0, shape = (1,), INIT_FROM_CKPT I0329 13:26:37.063537 10180 squad_utils.py:1637] name = end_logits/dense_1/bias:0, shape = (1,), INIT_FROM_CKPT INFO:tensorflow: name = answer_class/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT I0329 13:26:37.063537 10180 squad_utils.py:1637] name = answer_class/dense_0/kernel:0, shape = (1536, 768), INIT_FROM_CKPT INFO:tensorflow: name = answer_class/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT I0329 13:26:37.063537 10180 squad_utils.py:1637] name = answer_class/dense_0/bias:0, shape = (768,), INIT_FROM_CKPT INFO:tensorflow: name = answer_class/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT I0329 13:26:37.064536 10180 squad_utils.py:1637] name = answer_class/dense_1/kernel:0, shape = (768, 1), INIT_FROM_CKPT INFO:tensorflow:Done calling model_fn. I0329 13:26:37.064536 10180 estimator.py:1150] Done calling model_fn. WARNING:tensorflow:From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0329 13:26:37.093492 10180 deprecation.py:323] From C:\Users\alexa\anaconda3\envs\dynamic_QA\lib\site-packages\tensorflow_core\python\ops\array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where INFO:tensorflow:Graph was finalized. I0329 13:26:37.152336 10180 monitored_session.py:240] Graph was finalized. 2021-03-29 13:26:37.154873: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2021-03-29 13:26:37.158136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll 2021-03-29 13:26:37.168417: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error 2021-03-29 13:26:37.172695: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-GV232QR 2021-03-29 13:26:37.172943: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-GV232QR INFO:tensorflow:Restoring parameters from albert/output\model.ckpt-best I0329 13:26:37.177373 10180 saver.py:1284] Restoring parameters from albert/output\model.ckpt-best INFO:tensorflow:Running local_init_op. I0329 13:26:37.463492 10180 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0329 13:26:37.531100 10180 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Processing example: 0 I0329 13:26:42.834813 10180 run_squad_v2.py:389] Processing example: 0 INFO:tensorflow:prediction_loop marked as finished I0329 13:26:42.883660 10180 error_handling.py:101] prediction_loop marked as finished INFO:tensorflow:prediction_loop marked as finished I0329 13:26:42.884687 10180 error_handling.py:101] prediction_loop marked as finished INFO:tensorflow:Writing predictions to: albert/output\predictions.json I0329 13:26:42.886681 10180 squad_utils.py:1309] Writing predictions to: albert/output\predictions.json INFO:tensorflow:Writing nbest to: albert/output\nbest_predictions.json I0329 13:26:42.886681 10180 squad_utils.py:1310] Writing nbest to: albert/output\nbest_predictions.json INFO:tensorflow:Writing predictions to: albert/output\predictions.json I0329 13:26:42.908359 10180 squad_utils.py:1309] Writing predictions to: albert/output\predictions.json INFO:tensorflow:Writing nbest to: albert/output\nbest_predictions.json I0329 13:26:42.909356 10180 squad_utils.py:1310] Writing nbest to: albert/output\nbest_predictions.json INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.931493 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.932229 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.932597 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.933382 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.933740 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.933740 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.934739 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.934739 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.938729 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.938729 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.939726 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.939726 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.940616 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.941466 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.941834 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files. I0329 13:26:42.942656 10180 run_squad_v2.py:464] found 1 files. INFO:tensorflow:Add albert/output\model.ckpt-8144 to eval list. I0329 13:26:42.943008 10180 run_squad_v2.py:462] Add albert/output\model.ckpt-8144 to eval list. INFO:tensorflow:found 1 files.... this goes on indefinitely.