Mixed Precision issue when running with GTX 1080

[arashdehghani]

Hello, Good time I want to run inference using pre trained checkpoints for ASR models (jasper_10x3_8gpus_mp). But i have problem with 'dtype' !! this pretrained checkpoints have been trained with dtype='mixed' and my GPUs (GTX 1080) don't support Mixed precision training method!! when i change the dtype parameter to tf.float32 in config file, it can't restore checkpoint!! and when running with dtype=mixed it takes extra time to get inference results (about x3). have you any suggestion for this problem? can i convert checkpoint dtypes to tf.float32 before restoring it? thanks a lot Arash Dehghani

[borisgin]

Can you attach log for tf.float32 case please?

[arashdehghani]

Restoring from the latest checkpoint Loading model from jasper_10x3/w2lp34_checkpoint/model.ckpt-219600 Inference config: {'batch_size_per_gpu': 1, 'data_layer': <class 'open_seq2seq.data.speech2text.speech2text.Speech2TextDataLayer'>, 'data_layer_params': {'dataset_files': ['open_seq2seq/test_utils/toy_speech_data/toy_data_copy.csv'], 'input_type': 'logfbank', 'num_audio_features': 64, 'shuffle': False, 'vocab_file': 'open_seq2seq/test_utils/toy_speech_data/vocab.txt'}, 'decoder': <class 'open_seq2seq.decoders.fc_decoders.FullyConnectedCTCDecoder'>, 'decoder_params': {'alpha': 2.0, 'alphabet_config_path': 'open_seq2seq/test_utils/toy_speech_data/vocab.txt', 'beam_width': 512, 'beta': 1.5, 'decoder_library_path': 'ctc_decoder_with_lm/libctc_decoder_with_kenlm.so', 'initializer': <function xavier_initializer at 0x7f035d7b7268>, 'lm_path': 'language_model/4-gram.binary', 'trie_path': 'language_model/trie.binary', 'use_language_model': False}, 'dtype': tf.float32, 'encoder': <class 'open_seq2seq.encoders.tdnn_encoder.TDNNEncoder'>, 'encoder_params': {'activation_fn': <function at 0x7f0373e658c8>, 'convnet_layers': [{'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [11], 'num_channels': 256, 'padding': 'SAME', 'repeat': 1, 'stride': [2], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [11], 'num_channels': 256, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [11], 'num_channels': 256, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [13], 'num_channels': 384, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [13], 'num_channels': 384, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [17], 'num_channels': 512, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.8, 'kernel_size': [17], 'num_channels': 512, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.7, 'kernel_size': [21], 'num_channels': 640, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.7, 'kernel_size': [21], 'num_channels': 640, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.7, 'kernel_size': [25], 'num_channels': 768, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.7, 'kernel_size': [25], 'num_channels': 768, 'padding': 'SAME', 'repeat': 3, 'residual': True, 'stride': [1], 'type': 'conv1d'}, {'dilation': [2], 'dropout_keep_prob': 0.6, 'kernel_size': [29], 'num_channels': 896, 'padding': 'SAME', 'repeat': 1, 'stride': [1], 'type': 'conv1d'}, {'dilation': [1], 'dropout_keep_prob': 0.6, 'kernel_size': [1], 'num_channels': 1024, 'padding': 'SAME', 'repeat': 1, 'stride': [1], 'type': 'conv1d'}], 'data_format': 'channels_last', 'dropout_keep_prob': 0.7, 'initializer': <function xavier_initializer at 0x7f035d7b7268>, 'initializer_params': {'uniform': False}, 'normalization': 'batch_norm'}, 'eval_steps': 2200, 'iter_size': 1, 'larc_params': {'larc_eta': 0.001}, 'load_model': '', 'logdir': 'jasper_10x3/w2lp34_checkpoint', 'loss': <class 'open_seq2seq.losses.ctc_loss.CTCLoss'>, 'loss_params': {}, 'loss_scaling': 'Backoff', 'lr_policy': <function poly_decay at 0x7f035a636378>, 'lr_policy_params': {'learning_rate': 0.01, 'min_lr': 1e-05, 'power': 2.0}, 'num_epochs': 400, 'num_gpus': 1, 'optimizer': 'Momentum', 'optimizer_params': {'momentum': 0.9}, 'print_loss_steps': 10, 'print_samples_steps': 2200, 'random_seed': 0, 'regularizer': <function l2_regularizer at 0x7f035d828e18>, 'regularizer_params': {'scale': 0.001}, 'save_checkpoint_steps': 1100, 'save_summaries_steps': 100, 'summaries': ['learning_rate', 'variables', 'gradients', 'larc_summaries', 'variable_norm', 'gradient_norm', 'global_gradient_norm'], 'use_horovod': False, 'use_xla_jit': False} Building graph on GPU:0 WARNING:tensorflow:From /home/arman/Projects/ASR/ASR_English/open_seq2seq/data/speech2text/speech2text.py:267: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, use tf.py_function, which takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means tf.py_functions can use accelerators such as GPUs as well as being differentiable using a gradient tape.

WARNING:tensorflow:From /home/arman/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:1419: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /home/arman/Projects/ASR/ASR_English/open_seq2seq/parts/cnns/conv_blocks.py:159: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv1d instead. WARNING:tensorflow:From /home/arman/Projects/ASR/ASR_English/open_seq2seq/parts/cnns/conv_blocks.py:177: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.batch_normalization instead. WARNING:tensorflow:From /home/arman/Projects/ASR/ASR_English/open_seq2seq/encoders/tdnn_encoder.py:255: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. WARNING:tensorflow:From /home/arman/Projects/ASR/ASR_English/open_seq2seq/decoders/fc_decoders.py:139: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. ** Inference Mode. Loss part of graph isn't built. 2019-06-22 11:34:09.981704: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3311355000 Hz 2019-06-22 11:34:09.982076: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2488ef0 executing computations on platform Host. Devices: 2019-06-22 11:34:09.982100: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , WARNING:tensorflow:From /home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. 2019-06-22 11:34:10.198201: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Invalid argument: tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/bias; expected dtype float does not equal original dtype half tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv101/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv102/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv11/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv111/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv112/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv121/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv131/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv21/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv22/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv31/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv32/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv41/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv42/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv51/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv52/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv61/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv62/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv71/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv72/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv81/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv82/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv91/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv92/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/res/kernel; expected dtype float does not equal original dtype half Traceback (most recent call last): File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(args) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/bias; expected dtype float does not equal original dtype half tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv101/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv102/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv11/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv111/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv112/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv121/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv131/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv21/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv22/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv31/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv32/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv41/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv42/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv51/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv52/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv61/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv62/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv71/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv72/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv81/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv82/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv91/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv92/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/res/kernel; expected dtype float does not equal original dtype half [[{{node save/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1276, in restore {self.saver_def.filename_tensor_name: save_path}) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/bias; expected dtype float does not equal original dtype half tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv101/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv102/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv11/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv111/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv112/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv121/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv131/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv21/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv22/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv31/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv32/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv41/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv42/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv51/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv52/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv61/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv62/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv71/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv72/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv81/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv82/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv91/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv92/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/res/kernel; expected dtype float does not equal original dtype half [[node save/RestoreV2 (defined at /home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py:209) ]]

Caused by op 'save/RestoreV2', defined at: File "/home/arman/Projects/ASR/ASR_English/main_cpu.py", line 41, in res = General_ASR(wave_path) File "/home/arman/Projects/ASR/ASR_English/main_cpu.py", line 37, in General_ASR result = main(command_line_args) File "/home/arman/Projects/ASR/ASR_English/run.py", line 93, in main res = infer(model, checkpoint, args.infer_output_file) File "/home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py", line 230, in infer results_per_batch = restore_and_get_results(model, checkpoint, mode="infer") File "/home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py", line 209, in restore_and_get_results saver = tf.train.Saver() File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 832, in init self.build() File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 844, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 881, in _build build_save=build_save, build_restore=build_restore) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal restore_sequentially, reshape) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps restore_sequentially) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2 name=name) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/bias; expected dtype float does not equal original dtype half tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv101/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv102/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv11/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv111/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv112/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv121/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv131/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv21/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv22/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv31/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv32/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv41/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv42/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv51/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv52/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv61/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv62/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv71/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv72/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv81/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv82/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv91/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv92/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/res/kernel; expected dtype float does not equal original dtype half [[node save/RestoreV2 (defined at /home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py:209) ]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/arman/Projects/ASR/ASR_English/main_cpu.py", line 41, in res = General_ASR(wave_path) File "/home/arman/Projects/ASR/ASR_English/main_cpu.py", line 37, in General_ASR result = main(command_line_args) File "/home/arman/Projects/ASR/ASR_English/run.py", line 93, in main res = infer(model, checkpoint, args.infer_output_file) File "/home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py", line 230, in infer results_per_batch = restore_and_get_results(model, checkpoint, mode="infer") File "/home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py", line 218, in restore_and_get_results saver.restore(sess, checkpoint) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1312, in restore err, "a mismatch between the current graph and the graph") tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/bias; expected dtype float does not equal original dtype half tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv101/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv102/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv11/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv111/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv112/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv121/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv131/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv21/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv22/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv31/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv32/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv41/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv42/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv51/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv52/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv61/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv62/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv71/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv72/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv81/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv82/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv91/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv92/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/res/kernel; expected dtype float does not equal original dtype half [[node save/RestoreV2 (defined at /home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py:209) ]]

Caused by op 'save/RestoreV2', defined at: File "/home/arman/Projects/ASR/ASR_English/main_cpu.py", line 41, in res = General_ASR(wave_path) File "/home/arman/Projects/ASR/ASR_English/main_cpu.py", line 37, in General_ASR result = main(command_line_args) File "/home/arman/Projects/ASR/ASR_English/run.py", line 93, in main res = infer(model, checkpoint, args.infer_output_file) File "/home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py", line 230, in infer results_per_batch = restore_and_get_results(model, checkpoint, mode="infer") File "/home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py", line 209, in restore_and_get_results saver = tf.train.Saver() File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 832, in init self.build() File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 844, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 881, in _build build_save=build_save, build_restore=build_restore) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal restore_sequentially, reshape) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps restore_sequentially) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2 name=name) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/arman/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/bias; expected dtype float does not equal original dtype half tensor_name = ForwardPass/fully_connected_ctc_decoder/fully_connected/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv101/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv102/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv103/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv11/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv111/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv112/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv113/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv121/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv131/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv21/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv22/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv23/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv31/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv32/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv33/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv41/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv42/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv43/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv51/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv52/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv53/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv61/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv62/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv63/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv71/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv72/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv73/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv81/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv82/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv83/res/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv91/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv92/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/kernel; expected dtype float does not equal original dtype half tensor_name = ForwardPass/w2l_encoder/conv93/res/kernel; expected dtype float does not equal original dtype half [[node save/RestoreV2 (defined at /home/arman/Projects/ASR/ASR_English/open_seq2seq/utils/funcs.py:209) ]]

Process finished with exit code 1

thanks @borisgin

[blisc]

Can you pull the latest master and try again?

[arashdehghani]

@blisc thank you it now works for me but it's inference time is much longer than previous version!! I ran both of them on a single machine with CPU cores and RAM=6. I think it's about lack of memory!

[blisc]

The conversion from float16 <-> float32 uses tensorflow's assign ops which does take a noticeable amount of time.