search

emedvedev / attention-ocr

A Tensorflow model for text recognition (CNN + seq2seq with visual attention) available as a Python package and compatible with Google Cloud ML Engine.
MIT License
1.07k stars 257 forks source link

Unable to export model after training #184

Open eric-silk opened 3 years ago

eric-silk commented 3 years ago

I'm attempting to train a custom aocr model on an internal dataset. I've labeled the data using a directory of images, and an annotation file as described in the README. This was converted to a dataset (training.tfrecords) and then trained according to the instructions (with some parameters to accomodate our data):

aocr train --max-prediction 20 --full-ascii --num-epoch 300 datasets/gen_text_train/training.tfrecords

This ran to completion, and I attempt to export it as shown in the README:

aocr export --format=frozengraph ./attempted_training

This produces a VERY long error:

Full Error WARNING:tensorflow:From /home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/__main__.py:20: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/__main__.py:20: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead. 2021-02-10 09:18:26.028615: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-02-10 09:18:26.038234: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3399905000 Hz 2021-02-10 09:18:26.038452: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56193ecbc7d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-02-10 09:18:26.038477: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-02-10 09:18:26,039 root INFO phase: export 2021-02-10 09:18:26,039 root INFO model_dir: ./checkpoints 2021-02-10 09:18:26,039 root INFO load_model: True 2021-02-10 09:18:26,039 root INFO output_dir: ./results 2021-02-10 09:18:26,039 root INFO steps_per_checkpoint: 0 2021-02-10 09:18:26,039 root INFO batch_size: 1 2021-02-10 09:18:26,040 root INFO learning_rate: 1.000000 2021-02-10 09:18:26,040 root INFO reg_val: 0 2021-02-10 09:18:26,040 root INFO max_gradient_norm: 5.000000 2021-02-10 09:18:26,040 root INFO clip_gradients: True 2021-02-10 09:18:26,040 root INFO max_image_width 160.000000 2021-02-10 09:18:26,040 root INFO max_prediction_length 8.000000 2021-02-10 09:18:26,040 root INFO channels: 1 2021-02-10 09:18:26,040 root INFO target_embedding_size: 10.000000 2021-02-10 09:18:26,040 root INFO attn_num_hidden: 128 2021-02-10 09:18:26,040 root INFO attn_num_layers: 2 2021-02-10 09:18:26,040 root INFO visualize: False 2021-02-10 09:18:27,582 root INFO Reading model parameters from ./checkpoints/model.ckpt-33936 2021-02-10 09:18:27.635972: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv1/W/Initializer/random_uniform/shape (Const) conv_conv1/W/Initializer/random_uniform/min (Const) conv_conv1/W/Initializer/random_uniform/max (Const) conv_conv1/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv1/W/Initializer/random_uniform/sub (Sub) conv_conv1/W/Initializer/random_uniform/mul (Mul) conv_conv1/W/Initializer/random_uniform (Add) conv_conv1/W (VariableV2) /device:GPU:0 conv_conv1/W/Assign (Assign) /device:GPU:0 conv_conv1/W/read (Identity) /device:GPU:0 save/Assign_5 (Assign) /device:GPU:0 2021-02-10 09:18:27.636123: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv2/W/Initializer/random_uniform/shape (Const) conv_conv2/W/Initializer/random_uniform/min (Const) conv_conv2/W/Initializer/random_uniform/max (Const) conv_conv2/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv2/W/Initializer/random_uniform/sub (Sub) conv_conv2/W/Initializer/random_uniform/mul (Mul) conv_conv2/W/Initializer/random_uniform (Add) conv_conv2/W (VariableV2) /device:GPU:0 conv_conv2/W/Assign (Assign) /device:GPU:0 conv_conv2/W/read (Identity) /device:GPU:0 save/Assign_6 (Assign) /device:GPU:0 2021-02-10 09:18:27.636214: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv3/W/Initializer/random_uniform/shape (Const) conv_conv3/W/Initializer/random_uniform/min (Const) conv_conv3/W/Initializer/random_uniform/max (Const) conv_conv3/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv3/W/Initializer/random_uniform/sub (Sub) conv_conv3/W/Initializer/random_uniform/mul (Mul) conv_conv3/W/Initializer/random_uniform (Add) conv_conv3/W (VariableV2) /device:GPU:0 conv_conv3/W/Assign (Assign) /device:GPU:0 conv_conv3/W/read (Identity) /device:GPU:0 save/Assign_11 (Assign) /device:GPU:0 2021-02-10 09:18:27.636282: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv3/BatchNorm/gamma/Initializer/ones (Const) conv_conv3/BatchNorm/gamma (VariableV2) /device:GPU:0 conv_conv3/BatchNorm/gamma/Assign (Assign) /device:GPU:0 conv_conv3/BatchNorm/gamma/read (Identity) /device:GPU:0 save/Assign_8 (Assign) /device:GPU:0 2021-02-10 09:18:27.636333: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv3/BatchNorm/beta/Initializer/zeros (Const) conv_conv3/BatchNorm/beta (VariableV2) /device:GPU:0 conv_conv3/BatchNorm/beta/Assign (Assign) /device:GPU:0 conv_conv3/BatchNorm/beta/read (Identity) /device:GPU:0 save/Assign_7 (Assign) /device:GPU:0 2021-02-10 09:18:27.636387: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv3/BatchNorm/moving_mean/Initializer/zeros (Const) conv_conv3/BatchNorm/moving_mean (VariableV2) /device:GPU:0 conv_conv3/BatchNorm/moving_mean/Assign (Assign) /device:GPU:0 conv_conv3/BatchNorm/moving_mean/read (Identity) /device:GPU:0 save/Assign_9 (Assign) /device:GPU:0 2021-02-10 09:18:27.636465: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv3/BatchNorm/moving_variance/Initializer/ones (Const) conv_conv3/BatchNorm/moving_variance (VariableV2) /device:GPU:0 conv_conv3/BatchNorm/moving_variance/Assign (Assign) /device:GPU:0 conv_conv3/BatchNorm/moving_variance/read (Identity) /device:GPU:0 save/Assign_10 (Assign) /device:GPU:0 2021-02-10 09:18:27.636590: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv4/W/Initializer/random_uniform/shape (Const) conv_conv4/W/Initializer/random_uniform/min (Const) conv_conv4/W/Initializer/random_uniform/max (Const) conv_conv4/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv4/W/Initializer/random_uniform/sub (Sub) conv_conv4/W/Initializer/random_uniform/mul (Mul) conv_conv4/W/Initializer/random_uniform (Add) conv_conv4/W (VariableV2) /device:GPU:0 conv_conv4/W/Assign (Assign) /device:GPU:0 conv_conv4/W/read (Identity) /device:GPU:0 save/Assign_12 (Assign) /device:GPU:0 2021-02-10 09:18:27.636745: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv5/W/Initializer/random_uniform/shape (Const) conv_conv5/W/Initializer/random_uniform/min (Const) conv_conv5/W/Initializer/random_uniform/max (Const) conv_conv5/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv5/W/Initializer/random_uniform/sub (Sub) conv_conv5/W/Initializer/random_uniform/mul (Mul) conv_conv5/W/Initializer/random_uniform (Add) conv_conv5/W (VariableV2) /device:GPU:0 conv_conv5/W/Assign (Assign) /device:GPU:0 conv_conv5/W/read (Identity) /device:GPU:0 save/Assign_17 (Assign) /device:GPU:0 2021-02-10 09:18:27.636882: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv5/BatchNorm/gamma/Initializer/ones (Const) conv_conv5/BatchNorm/gamma (VariableV2) /device:GPU:0 conv_conv5/BatchNorm/gamma/Assign (Assign) /device:GPU:0 conv_conv5/BatchNorm/gamma/read (Identity) /device:GPU:0 save/Assign_14 (Assign) /device:GPU:0 2021-02-10 09:18:27.636984: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv5/BatchNorm/beta/Initializer/zeros (Const) conv_conv5/BatchNorm/beta (VariableV2) /device:GPU:0 conv_conv5/BatchNorm/beta/Assign (Assign) /device:GPU:0 conv_conv5/BatchNorm/beta/read (Identity) /device:GPU:0 save/Assign_13 (Assign) /device:GPU:0 2021-02-10 09:18:27.637085: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv5/BatchNorm/moving_mean/Initializer/zeros (Const) conv_conv5/BatchNorm/moving_mean (VariableV2) /device:GPU:0 conv_conv5/BatchNorm/moving_mean/Assign (Assign) /device:GPU:0 conv_conv5/BatchNorm/moving_mean/read (Identity) /device:GPU:0 save/Assign_15 (Assign) /device:GPU:0 2021-02-10 09:18:27.637197: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv5/BatchNorm/moving_variance/Initializer/ones (Const) conv_conv5/BatchNorm/moving_variance (VariableV2) /device:GPU:0 conv_conv5/BatchNorm/moving_variance/Assign (Assign) /device:GPU:0 conv_conv5/BatchNorm/moving_variance/read (Identity) /device:GPU:0 save/Assign_16 (Assign) /device:GPU:0 2021-02-10 09:18:27.637320: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv6/W/Initializer/random_uniform/shape (Const) conv_conv6/W/Initializer/random_uniform/min (Const) conv_conv6/W/Initializer/random_uniform/max (Const) conv_conv6/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv6/W/Initializer/random_uniform/sub (Sub) conv_conv6/W/Initializer/random_uniform/mul (Mul) conv_conv6/W/Initializer/random_uniform (Add) conv_conv6/W (VariableV2) /device:GPU:0 conv_conv6/W/Assign (Assign) /device:GPU:0 conv_conv6/W/read (Identity) /device:GPU:0 save/Assign_18 (Assign) /device:GPU:0 2021-02-10 09:18:27.637472: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv7/W/Initializer/random_uniform/shape (Const) conv_conv7/W/Initializer/random_uniform/min (Const) conv_conv7/W/Initializer/random_uniform/max (Const) conv_conv7/W/Initializer/random_uniform/RandomUniform (RandomUniform) conv_conv7/W/Initializer/random_uniform/sub (Sub) conv_conv7/W/Initializer/random_uniform/mul (Mul) conv_conv7/W/Initializer/random_uniform (Add) conv_conv7/W (VariableV2) /device:GPU:0 conv_conv7/W/Assign (Assign) /device:GPU:0 conv_conv7/W/read (Identity) /device:GPU:0 save/Assign_23 (Assign) /device:GPU:0 2021-02-10 09:18:27.637608: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv7/BatchNorm/gamma/Initializer/ones (Const) conv_conv7/BatchNorm/gamma (VariableV2) /device:GPU:0 conv_conv7/BatchNorm/gamma/Assign (Assign) /device:GPU:0 conv_conv7/BatchNorm/gamma/read (Identity) /device:GPU:0 save/Assign_20 (Assign) /device:GPU:0 2021-02-10 09:18:27.638096: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv7/BatchNorm/beta/Initializer/zeros (Const) conv_conv7/BatchNorm/beta (VariableV2) /device:GPU:0 conv_conv7/BatchNorm/beta/Assign (Assign) /device:GPU:0 conv_conv7/BatchNorm/beta/read (Identity) /device:GPU:0 save/Assign_19 (Assign) /device:GPU:0 2021-02-10 09:18:27.638206: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv7/BatchNorm/moving_mean/Initializer/zeros (Const) conv_conv7/BatchNorm/moving_mean (VariableV2) /device:GPU:0 conv_conv7/BatchNorm/moving_mean/Assign (Assign) /device:GPU:0 conv_conv7/BatchNorm/moving_mean/read (Identity) /device:GPU:0 save/Assign_21 (Assign) /device:GPU:0 2021-02-10 09:18:27.638308: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: conv_conv7/BatchNorm/moving_variance/Initializer/ones (Const) conv_conv7/BatchNorm/moving_variance (VariableV2) /device:GPU:0 conv_conv7/BatchNorm/moving_variance/Assign (Assign) /device:GPU:0 conv_conv7/BatchNorm/moving_variance/read (Identity) /device:GPU:0 save/Assign_22 (Assign) /device:GPU:0 2021-02-10 09:18:27.638472: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/shape (Const) bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/min (Const) bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/max (Const) bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/sub (Sub) bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform/mul (Mul) bidirectional_rnn/fw/basic_lstm_cell/kernel/Initializer/random_uniform (Add) bidirectional_rnn/fw/basic_lstm_cell/kernel (VariableV2) /device:GPU:0 bidirectional_rnn/fw/basic_lstm_cell/kernel/Assign (Assign) /device:GPU:0 save/Assign_4 (Assign) /device:GPU:0 2021-02-10 09:18:27.638609: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU VariableV2: CPU Const: CPU XLA_CPU Fill: CPU XLA_CPU Colocation members, user-requested devices, and framework assigned devices, if any: bidirectional_rnn/fw/basic_lstm_cell/bias/Initializer/zeros/shape_as_tensor (Const) bidirectional_rnn/fw/basic_lstm_cell/bias/Initializer/zeros/Const (Const) bidirectional_rnn/fw/basic_lstm_cell/bias/Initializer/zeros (Fill) bidirectional_rnn/fw/basic_lstm_cell/bias (VariableV2) /device:GPU:0 bidirectional_rnn/fw/basic_lstm_cell/bias/Assign (Assign) /device:GPU:0 save/Assign_3 (Assign) /device:GPU:0 2021-02-10 09:18:27.639111: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/shape (Const) bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/min (Const) bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/max (Const) bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/sub (Sub) bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform/mul (Mul) bidirectional_rnn/bw/basic_lstm_cell/kernel/Initializer/random_uniform (Add) bidirectional_rnn/bw/basic_lstm_cell/kernel (VariableV2) /device:GPU:0 bidirectional_rnn/bw/basic_lstm_cell/kernel/Assign (Assign) /device:GPU:0 save/Assign_2 (Assign) /device:GPU:0 2021-02-10 09:18:27.639261: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU VariableV2: CPU Const: CPU XLA_CPU Fill: CPU XLA_CPU Colocation members, user-requested devices, and framework assigned devices, if any: bidirectional_rnn/bw/basic_lstm_cell/bias/Initializer/zeros/shape_as_tensor (Const) bidirectional_rnn/bw/basic_lstm_cell/bias/Initializer/zeros/Const (Const) bidirectional_rnn/bw/basic_lstm_cell/bias/Initializer/zeros (Fill) bidirectional_rnn/bw/basic_lstm_cell/bias (VariableV2) /device:GPU:0 bidirectional_rnn/bw/basic_lstm_cell/bias/Assign (Assign) /device:GPU:0 save/Assign_1 (Assign) /device:GPU:0 2021-02-10 09:18:27.639855: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/shape (Const) embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/min (Const) embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/max (Const) embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/RandomUniform (RandomUniform) embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/sub (Sub) embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform/mul (Mul) embedding_attention_decoder/attention_decoder/AttnW_0/Initializer/random_uniform (Add) embedding_attention_decoder/attention_decoder/AttnW_0 (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnW_0/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnW_0/read (Identity) /device:GPU:0 save/Assign_29 (Assign) /device:GPU:0 2021-02-10 09:18:27.640798: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/shape (Const) embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/min (Const) embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/max (Const) embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/RandomUniform (RandomUniform) embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/sub (Sub) embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform/mul (Mul) embedding_attention_decoder/attention_decoder/AttnV_0/Initializer/random_uniform (Add) embedding_attention_decoder/attention_decoder/AttnV_0 (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnV_0/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnV_0/read (Identity) /device:GPU:0 save/Assign_28 (Assign) /device:GPU:0 2021-02-10 09:18:27.641065: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/shape (Const) embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/min (Const) embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/max (Const) embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/sub (Sub) embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform/mul (Mul) embedding_attention_decoder/attention_decoder/kernel/Initializer/random_uniform (Add) embedding_attention_decoder/attention_decoder/kernel (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/kernel/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/kernel/read (Identity) /device:GPU:0 save/Assign_31 (Assign) /device:GPU:0 2021-02-10 09:18:27.641259: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/bias/Initializer/Const (Const) embedding_attention_decoder/attention_decoder/bias (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/bias/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/bias/read (Identity) /device:GPU:0 save/Assign_30 (Assign) /device:GPU:0 2021-02-10 09:18:27.641389: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/shape (Const) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/min (Const) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/max (Const) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/sub (Sub) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform/mul (Mul) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Initializer/random_uniform (Add) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/kernel/read (Identity) /device:GPU:0 save/Assign_33 (Assign) /device:GPU:0 2021-02-10 09:18:27.641530: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias/Initializer/zeros (Const) embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/basic_lstm_cell/bias/read (Identity) /device:GPU:0 save/Assign_32 (Assign) /device:GPU:0 2021-02-10 09:18:27.641686: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/shape (Const) embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/min (Const) embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/max (Const) embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/sub (Sub) embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform/mul (Mul) embedding_attention_decoder/attention_decoder/Attention_0/kernel/Initializer/random_uniform (Add) embedding_attention_decoder/attention_decoder/Attention_0/kernel (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/Attention_0/kernel/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/Attention_0/kernel/read (Identity) /device:GPU:0 save/Assign_25 (Assign) /device:GPU:0 2021-02-10 09:18:27.641934: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/Attention_0/bias/Initializer/Const (Const) embedding_attention_decoder/attention_decoder/Attention_0/bias (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/Attention_0/bias/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/Attention_0/bias/read (Identity) /device:GPU:0 save/Assign_24 (Assign) /device:GPU:0 2021-02-10 09:18:27.642040: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU RandomUniform: CPU XLA_CPU Const: CPU XLA_CPU Mul: CPU XLA_CPU Sub: CPU XLA_CPU Add: CPU XLA_CPU Identity: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/shape (Const) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/min (Const) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/max (Const) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/sub (Sub) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform/mul (Mul) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Initializer/random_uniform (Add) embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnOutputProjection/kernel/read (Identity) /device:GPU:0 save/Assign_27 (Assign) /device:GPU:0 2021-02-10 09:18:27.642139: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: CPU XLA_CPU Assign: CPU Const: CPU XLA_CPU VariableV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias/Initializer/Const (Const) embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias (VariableV2) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias/Assign (Assign) /device:GPU:0 embedding_attention_decoder/attention_decoder/AttnOutputProjection/bias/read (Identity) /device:GPU:0 save/Assign_26 (Assign) /device:GPU:0 2021-02-10 09:18:27.642847: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Switch: CPU XLA_CPU Enter: CPU XLA_CPU LookupTableFindV2: CPU LookupTableInsertV2: CPU MutableHashTableV2: CPU LookupTableExportV2: CPU Colocation members, user-requested devices, and framework assigned devices, if any: MutableHashTable (MutableHashTableV2) /device:GPU:0 MutableHashTable_lookup_table_export_values/LookupTableExportV2 (LookupTableExportV2) /device:GPU:0 MutableHashTable_lookup_table_insert/LookupTableInsertV2 (LookupTableInsertV2) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter (Enter) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter_1 (Enter) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Switch (Switch) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter_2 (Enter) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Enter_3 (Enter) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2/Switch_2 (Switch) /device:GPU:0 map_1/while/foldr/while/cond/MutableHashTable_lookup_table_find/LookupTableFindV2 (LookupTableFindV2) /device:GPU:0 Traceback (most recent call last): File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10] [[{{node save/Assign_34}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore {self.saver_def.filename_tensor_name: save_path}) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10] [[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] Original stack trace for 'save/Assign_34': File "/bin/aocr", line 8, in sys.exit(main()) File "/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main channels=parameters.channels, File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in __init__ self.saver_all = tf.train.Saver(tf.all_variables()) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in __init__ self.build() File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build self._build(self._filename, build_save=True, build_restore=True) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build build_restore=build_restore) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal restore_sequentially, reshape) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore self.op.get_shape().is_fully_defined()) File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign validate_shape=validate_shape) File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign use_locking=use_locking, name=name) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ericsilk/anaconda3/bin/aocr", line 8, in sys.exit(main()) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main channels=parameters.channels, File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/model/model.py", line 268, in __init__ self.saver_all.restore(self.sess, ckpt.model_checkpoint_path) File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1326, in restore err, "a mismatch between the current graph and the graph") tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10] [[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] Original stack trace for 'save/Assign_34': File "/bin/aocr", line 8, in sys.exit(main()) File "/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main channels=parameters.channels, File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in __init__ self.saver_all = tf.train.Saver(tf.all_variables()) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in __init__ self.build() File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build self._build(self._filename, build_save=True, build_restore=True) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build build_restore=build_restore) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal restore_sequentially, reshape) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore self.op.get_shape().is_fully_defined()) File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign validate_shape=validate_shape) File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign use_locking=use_locking, name=name) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack()}}

The key bits of which appear to be:

Traceback (most recent call last):
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
         [[{{node save/Assign_34}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
         [[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/Assign_34':
  File "/bin/aocr", line 8, in <module>
    sys.exit(main())
  File "/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main
    channels=parameters.channels,
  File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in __init__
    self.saver_all = tf.train.Saver(tf.all_variables())
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in __init__
    self.build()
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
    self.op.get_shape().is_fully_defined())
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ericsilk/anaconda3/bin/aocr", line 8, in <module>
    sys.exit(main())
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main
    channels=parameters.channels,
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/aocr/model/model.py", line 268, in __init__
    self.saver_all.restore(self.sess, ckpt.model_checkpoint_path)
  File "/home/ericsilk/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1326, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [39,10] rhs shape= [98,10]
         [[node save/Assign_34 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/Assign_34':
  File "/bin/aocr", line 8, in <module>
    sys.exit(main())
  File "/lib/python3.7/site-packages/aocr/__main__.py", line 251, in main
    channels=parameters.channels,
  File "/lib/python3.7/site-packages/aocr/model/model.py", line 261, in __init__
    self.saver_all = tf.train.Saver(tf.all_variables())
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in __init__
    self.build()
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
    self.op.get_shape().is_fully_defined())
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

System info: Ubuntu 18.04.5 LTS, Anaconda 3 Any suggestions?

maxpaynestory commented 3 years ago

I frequently use aocr inside Google Colabs and i haven't got any problems. I use tensorflow_gpu==1.15 with aocr