search

NVIDIA / OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
https://nvidia.github.io/OpenSeq2Seq
Apache License 2.0
1.54k stars 371 forks source link

tensorflow/core/util/ctc/ctc_loss_calculator.cc:144] No valid path found. while loading the model. #453

Closed pratapaprasanna closed 5 years ago

pratapaprasanna commented 5 years ago

HI all ,

I have trained the Openseq2seq speech2Text (jasper10x5_LibriSpeech_nvgrad.py) for two days and predictions went well.

Now i want to load this model and fire another training for which i have done only the following.

ls /home/ubuntu/backup_logs/checkpoint_dir/

best_models                                     graph.pbtxt                           model.ckpt-56100.meta
checkpoint                                      model.ckpt-29700.data-00000-of-00001  model.ckpt-57200.data-00000-of-00001
events.out.tfevents.1559164634.ip-172-31-0-118  model.ckpt-29700.index                model.ckpt-57200.index
events.out.tfevents.1559236856.ip-172-31-0-118  model.ckpt-29700.meta                 model.ckpt-57200.meta
events.out.tfevents.1559384269.ip-172-31-0-118  model.ckpt-56100.data-00000-of-00001
events.out.tfevents.1559385631.ip-172-31-0-118  model.ckpt-56100.index

     "save_summaries_steps": 100,
     "load_model": "/home/ubuntu/backup_logs/checkpoint_dir",
     "logdir": "/home/ubuntu/backup_logs/checkpoint_dir",
     "print_loss_steps": 10,
     "print_samples_steps": 2200,
     "eval_steps": 2200,

and i triggered training on the same dataset and now i am getting this in my logs

Following is the log:

ubuntu@ip-172-31-0-118:~/OpenSeq2Seq$ cat nohup.out

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

*** Restored checkpoint from /home/ubuntu/backup_logs/dhwani_log/model.ckpt-57200. Resuming training
*** Training config:
{'batch_size_per_gpu': 64,
 'data_layer': <class 'open_seq2seq.data.speech2text.speech2text.Speech2TextDataLayer'>,
 'data_layer_params': {'augmentation': {'speed_perturbation_ratio': [0.9,
                                                                     1.0,
                                                                     1.1]},
                       'backend': 'librosa',
                       'dataset_files': ['/home/ubuntu/train_jasper_new_mapping.csv'],
                       'dither': 1e-05,
                       'input_type': 'logfbank',
                       'max_duration': 16.7,
                       'norm_per_feature': True,
                       'num_audio_features': 64,
                       'pad_to': 16,
                       'precompute_mel_basis': True,
                       'sample_freq': 8000,
                       'shuffle': True,
                       'vocab_file': 'open_seq2seq/test_utils/toy_speech_data/vocab.txt',
                       'window': 'hanning'},
 'decoder': <class 'open_seq2seq.decoders.fc_decoders.FullyConnectedCTCDecoder'>,
 'decoder_params': {'infer_logits_to_pickle': False,
                    'initializer': <function xavier_initializer at 0x7fe7e5487158>,
                    'use_language_model': False},
 'dtype': 'mixed',
 'encoder': <class 'open_seq2seq.encoders.tdnn_encoder.TDNNEncoder'>,
 'encoder_params': {'activation_fn': <function relu at 0x7fe764201840>,
                    'convnet_layers': [{'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [11],
                                        'num_channels': 256,
                                        'padding': 'SAME',
                                        'repeat': 1,
                                        'stride': [2],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [11],
                                        'num_channels': 256,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [11],
                                        'num_channels': 256,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [13],
                                        'num_channels': 384,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [13],
                                        'num_channels': 384,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [17],
                                        'num_channels': 512,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [17],
                                        'num_channels': 512,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [21],
                                        'num_channels': 640,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [21],
                                        'num_channels': 640,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [25],
                                        'num_channels': 768,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [25],
                                        'num_channels': 768,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [2],
                                        'dropout_keep_prob': 0.6,
                                        'kernel_size': [29],
                                        'num_channels': 896,
                                        'padding': 'SAME',
                                        'repeat': 1,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.6,
                                        'kernel_size': [1],
                                        'num_channels': 1024,
                                        'padding': 'SAME',
                                        'repeat': 1,
                                        'stride': [1],
                                        'type': 'conv1d'}],
                    'data_format': 'channels_last',
                    'dropout_keep_prob': 0.7,
                    'initializer': <function xavier_initializer at 0x7fe7e5487158>,
                    'initializer_params': {'uniform': False},
                    'normalization': 'batch_norm',
                    'use_conv_mask': True},
 'eval_steps': 2200,
 'iter_size': 1,
 'larc_params': {'larc_eta': 0.001},
 'load_model': '/home/ubuntu/backup_logs/checkpoint_dir,
 'logdir': '/home/ubuntu/backup_logs/checkpoint_dir,
 'loss': <class 'open_seq2seq.losses.ctc_loss.CTCLoss'>,
 'loss_params': {},
 'loss_scaling': 'Backoff',
 'lr_policy': <function poly_decay at 0x7fe758106d08>,
 'lr_policy_params': {'learning_rate': 0.02, 'min_lr': 1e-05, 'power': 2.0},
 'num_checkpoints': 2,
 'num_epochs': 400,
 'num_gpus': 8,
 'optimizer': <class 'open_seq2seq.optimizers.novograd.NovoGrad'>,
 'optimizer_params': {'beta1': 0.95,
                      'beta2': 0.98,
                      'epsilon': 1e-08,
                      'grad_averaging': False,
                      'weight_decay': 0.001},
 'print_loss_steps': 10,
 'print_samples_steps': 2200,
 'random_seed': 0,
 'save_checkpoint_steps': 1100,
 'save_summaries_steps': 100,
 'summaries': ['learning_rate',
               'variables',
               'gradients',
               'larc_summaries',
               'variable_norm',
               'gradient_norm',
               'global_gradient_norm'],
 'use_horovod': False,
 'use_xla_jit': False}
*** Evaluation config:
{'batch_size_per_gpu': 64,
 'data_layer': <class 'open_seq2seq.data.speech2text.speech2text.Speech2TextDataLayer'>,
 'data_layer_params': {'backend': 'librosa',
                       'dataset_files': ['/home/ubuntu/Manifest/test_v1.csv'],
                       'dither': 1e-05,
                       'input_type': 'logfbank',
                       'norm_per_feature': True,
                       'num_audio_features': 64,
                       'pad_to': 16,
                       'precompute_mel_basis': True,
                       'sample_freq': 8000,
                       'shuffle': False,
                       'vocab_file': 'open_seq2seq/test_utils/toy_speech_data/vocab.txt',
                       'window': 'hanning'},
 'decoder': <class 'open_seq2seq.decoders.fc_decoders.FullyConnectedCTCDecoder'>,
 'decoder_params': {'infer_logits_to_pickle': False,
                    'initializer': <function xavier_initializer at 0x7fe7e5487158>,
                    'use_language_model': False},
 'dtype': 'mixed',
 'encoder': <class 'open_seq2seq.encoders.tdnn_encoder.TDNNEncoder'>,
 'encoder_params': {'activation_fn': <function relu at 0x7fe764201840>,
                    'convnet_layers': [{'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [11],
                                        'num_channels': 256,
                                        'padding': 'SAME',
                                        'repeat': 1,
                                        'stride': [2],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [11],
                                        'num_channels': 256,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [11],
                                        'num_channels': 256,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [13],
                                        'num_channels': 384,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [13],
                                        'num_channels': 384,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [17],
                                        'num_channels': 512,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.8,
                                        'kernel_size': [17],
                                        'num_channels': 512,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [21],
                                        'num_channels': 640,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense'WARNING:tensorflow:From /home/ubuntu/OpenSeq2Seq/open_seq2seq/data/speech2text/speech2text.py:216: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
    tf.py_function, which takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.

WARNING:tensorflow:From /home/ubuntu/myenv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:1419: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/ubuntu/OpenSeq2Seq/open_seq2seq/parts/cnns/conv_blocks.py:159: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv1d instead.
WARNING:tensorflow:From /home/ubuntu/OpenSeq2Seq/open_seq2seq/parts/cnns/conv_blocks.py:177: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.batch_normalization instead.
WARNING:tensorflow:From /home/ubuntu/OpenSeq2Seq/open_seq2seq/encoders/tdnn_encoder.py:255: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/ubuntu/OpenSeq2Seq/open_seq2seq/decoders/fc_decoders.py:139: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
WARNING:tensorflow:From /home/ubuntu/myenv/lib/python3.6/site-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
: True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [21],
                                        'num_channels': 640,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [25],
                                        'num_channels': 768,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.7,
                                        'kernel_size': [25],
                                        'num_channels': 768,
                                        'padding': 'SAME',
                                        'repeat': 5,
                                        'residual': True,
                                        'residual_dense': True,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [2],
                                        'dropout_keep_prob': 0.6,
                                        'kernel_size': [29],
                                        'num_channels': 896,
                                        'padding': 'SAME',
                                        'repeat': 1,
                                        'stride': [1],
                                        'type': 'conv1d'},
                                       {'dilation': [1],
                                        'dropout_keep_prob': 0.6,
                                        'kernel_size': [1],
                                        'num_channels': 1024,
                                        'padding': 'SAME',
                                        'repeat': 1,
                                        'stride': [1],
                                        'type': 'conv1d'}],
                    'data_format': 'channels_last',
                    'dropout_keep_prob': 0.7,
                    'initializer': <function xavier_initializer at 0x7fe7e5487158>,
                    'initializer_params': {'uniform': False},
                    'normalization': 'batch_norm',
                    'use_conv_mask': True},
 'eval_steps': 2200,
 'iter_size': 1,
 'larc_params': {'larc_eta': 0.001},
 'load_model': '/home/ubuntu/backup_logs/dhwani_log',
 'logdir': '/home/ubuntu/backup_logs/dhwani_log',
 'loss': <class 'open_seq2seq.losses.ctc_loss.CTCLoss'>,
 'loss_params': {},
 'loss_scaling': 'Backoff',
 'lr_policy': <function poly_decay at 0x7fe758106d08>,
 'lr_policy_params': {'learning_rate': 0.02, 'min_lr': 1e-05, 'power': 2.0},
 'num_checkpoints': 2,
 'num_epochs': 400,
 'num_gpus': 8,
 'optimizer': <class 'open_seq2seq.optimizers.novograd.NovoGrad'>,
 'optimizer_params': {'beta1': 0.95,
                      'beta2': 0.98,
                      'epsilon': 1e-08,
                      'grad_averaging': False,
                      'weight_decay': 0.001},
 'print_loss_steps': 10,
 'print_samples_steps': 2200,
 'random_seed': 0,
 'save_checkpoint_steps': 1100,
 'save_summaries_steps': 100,
 'summaries': ['learning_rate',
               'variables',
               'gradients',
               'larc_summaries',
               'variable_norm',
               'gradient_norm',
               'global_gradient_norm'],
 'use_horovod': False,
 'use_xla_jit': False}
*** Building graph on GPU:0
*** Building graph on GPU:1
*** Building graph on GPU:2
*** Building graph on GPU:3
*** Building graph on GPU:4
*** Building graph on GPU:5
*** Building graph on GPU:6
*** Building graph on GPU:7
*** Trainable variables:
***   ForwardPass/w2l_encoder/conv11/kernel:0
***     shape: (11, 64, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv11/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv11/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv21/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv21/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv21/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv22/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv22/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv22/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv23/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv23/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv23/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv24/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv24/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv24/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv25/res_0/kernel:0
***     shape: (1, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv25/res_bn_0/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv25/res_bn_0/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv25/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv25/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv25/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv31/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv31/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv31/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv32/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv32/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv32/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv33/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv33/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv33/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv34/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv34/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv34/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv35/res_0/kernel:0
***     shape: (1, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv35/res_bn_0/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv35/res_bn_0/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv35/res_1/kernel:0
***     shape: (1, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv35/res_bn_1/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv35/res_bn_1/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv35/kernel:0
***     shape: (11, 256, 256), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv35/bn/gamma:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv35/bn/beta:0
***     shape: (256,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv41/kernel:0
***     shape: (13, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv41/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv41/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv42/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv42/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv42/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv43/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv43/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv43/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv44/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv44/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv44/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/res_0/kernel:0
***     shape: (1, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv45/res_bn_0/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/res_bn_0/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/res_1/kernel:0
***     shape: (1, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv45/res_bn_1/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/res_bn_1/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/res_2/kernel:0
***     shape: (1, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv45/res_bn_2/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/res_bn_2/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv45/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv45/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv51/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv51/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv51/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv52/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv52/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv52/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv53/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv53/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv53/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv54/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv54/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv54/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_0/kernel:0
***     shape: (1, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_0/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_0/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_1/kernel:0
***     shape: (1, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_1/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_1/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_2/kernel:0
***     shape: (1, 256, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_2/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_2/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_3/kernel:0
***     shape: (1, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_3/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/res_bn_3/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/kernel:0
***     shape: (13, 384, 384), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv55/bn/gamma:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv55/bn/beta:0
***     shape: (384,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv61/kernel:0
***     shape: (17, 384, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv61/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv61/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv62/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv62/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv62/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv63/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv63/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv63/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv64/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv64/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv64/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_0/kernel:0
***     shape: (1, 256, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_0/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_0/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_1/kernel:0
***     shape: (1, 256, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_1/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_1/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_2/kernel:0
***     shape: (1, 256, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_2/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_2/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_3/kernel:0
***     shape: (1, 384, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_3/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_3/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_4/kernel:0
***     shape: (1, 384, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_4/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/res_bn_4/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv65/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv65/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv71/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv71/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv71/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv72/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv72/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv72/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv73/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv73/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv73/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv74/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv74/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv74/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_0/kernel:0
***     shape: (1, 256, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_0/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_0/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_1/kernel:0
***     shape: (1, 256, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_1/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_1/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_2/kernel:0
***     shape: (1, 256, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_2/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_2/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_3/kernel:0
***     shape: (1, 384, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_3/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_3/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_4/kernel:0
***     shape: (1, 384, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_4/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_4/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_5/kernel:0
***     shape: (1, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_5/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/res_bn_5/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/kernel:0
***     shape: (17, 512, 512), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv75/bn/gamma:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv75/bn/beta:0
***     shape: (512,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv81/kernel:0
***     shape: (21, 512, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv81/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv81/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv82/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv82/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv82/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv83/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv83/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv83/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv84/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv84/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv84/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_0/kernel:0
***     shape: (1, 256, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_0/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_0/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_1/kernel:0
***     shape: (1, 256, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_1/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_1/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_2/kernel:0
***     shape: (1, 256, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_2/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_2/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_3/kernel:0
***     shape: (1, 384, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_3/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_3/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_4/kernel:0
***     shape: (1, 384, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_4/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_4/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_5/kernel:0
***     shape: (1, 512, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_5/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_5/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_6/kernel:0
***     shape: (1, 512, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_6/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/res_bn_6/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv85/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv85/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv91/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv91/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv91/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv92/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv92/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv92/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv93/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv93/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv93/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv94/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv94/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv94/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_0/kernel:0
***     shape: (1, 256, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_0/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_0/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_1/kernel:0
***     shape: (1, 256, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_1/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_1/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_2/kernel:0
***     shape: (1, 256, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_2/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_2/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_3/kernel:0
***     shape: (1, 384, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_3/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_3/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_4/kernel:0
***     shape: (1, 384, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_4/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_4/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_5/kernel:0
***     shape: (1, 512, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_5/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_5/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_6/kernel:0
***     shape: (1, 512, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_6/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_6/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_7/kernel:0
***     shape: (1, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_7/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/res_bn_7/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/kernel:0
***     shape: (21, 640, 640), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv95/bn/gamma:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv95/bn/beta:0
***     shape: (640,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv101/kernel:0
***     shape: (25, 640, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv101/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv101/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv102/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv102/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv102/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv103/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv103/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv103/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv104/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv104/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv104/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_0/kernel:0
***     shape: (1, 256, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_0/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_0/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_1/kernel:0
***     shape: (1, 256, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_1/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_1/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_2/kernel:0
***     shape: (1, 256, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_2/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_2/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_3/kernel:0
***     shape: (1, 384, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_3/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_3/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_4/kernel:0
***     shape: (1, 384, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_4/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_4/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_5/kernel:0
***     shape: (1, 512, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_5/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_5/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_6/kernel:0
***     shape: (1, 512, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_6/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_6/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_7/kernel:0
***     shape: (1, 640, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_7/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_7/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_8/kernel:0
***     shape: (1, 640, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_8/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/res_bn_8/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv105/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv105/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv111/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv111/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv111/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv112/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv112/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv112/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv113/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv113/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv113/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv114/kernel:0
***     shape: (25, 768, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv114/bn/gamma:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv114/bn/beta:0
***     shape: (768,), <dtype: 'float32_ref'>
***   ForwardPass/w2l_encoder/conv115/res_0/kernel:0
***     shape: (1, 256, 768), <dtype: 'float16_ref'>
***   ForwardPass/w2l_encoder/conv115/res_bn_0/gamma:02019-06-01 10:40:41.853766: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-06-01 10:40:46.815724: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.884758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.890800: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.914150: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.929368: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.945161: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.962910: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.983473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-01 10:40:46.985058: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x7b0a6f10 executing computations on platform CUDA. Devices:
2019-06-01 10:40:46.985094: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985102: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (1): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985107: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (2): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985128: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (3): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985133: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (4): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985138: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (5): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985153: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (6): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:46.985158: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (7): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-06-01 10:40:47.009158: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500000000 Hz
2019-06-01 10:40:47.014975: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5fae10b0 executing computations on platform Host. Devices:
2019-06-01 10:40:47.015003: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-06-01 10:40:47.015509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:16.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.015802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:17.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.016071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 2 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:18.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.016349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 3 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:19.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 4 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1a.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.016913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 5 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1b.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.017191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 6 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1c.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.017489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 7 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1d.0
totalMemory: 31.72GiB freeMemory: 31.31GiB
2019-06-01 10:40:47.017810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7
2019-06-01 10:40:47.027654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-06-01 10:40:47.027676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 1 2 3 4 5 6 7
2019-06-01 10:40:47.027683: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N Y Y Y Y N N N
2019-06-01 10:40:47.027688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1:   Y N Y Y N Y N N
2019-06-01 10:40:47.027694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2:   Y Y N Y N N Y N
2019-06-01 10:40:47.027698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3:   Y Y Y N N N N Y
2019-06-01 10:40:47.027703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 4:   Y N N N N Y Y Y
2019-06-01 10:40:47.027708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 5:   N Y N N Y N Y Y
2019-06-01 10:40:47.027714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 6:   N N Y N Y Y N Y
2019-06-01 10:40:47.027719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 7:   N N N Y Y Y Y N
2019-06-01 10:40:47.029866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30458 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:16.0, compute capability: 7.0)
2019-06-01 10:40:47.030217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30458 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:17.0, compute capability: 7.0)
2019-06-01 10:40:47.030583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30458 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:18.0, compute capability: 7.0)
2019-06-01 10:40:47.030789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30458 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:19.0, compute capability: 7.0)
2019-06-01 10:40:47.031114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 30458 MB memory) -> physical GPU (device: 4, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:1a.0, compute capability: 7.0)
2019-06-01 10:40:47.031436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 30458 MB memory) -> physical GPU (device: 5, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:1b.0, compute capability: 7.0)
2019-06-01 10:40:47.031722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:6 with 30458 MB memory) -> physical GPU (device: 6, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:1c.0, compute capability: 7.0)
2019-06-01 10:40:47.032010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:7 with 30458 MB memory) -> physical GPU (device: 7, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:1d.0, compute capability: 7.0)
WARNING:tensorflow:From /home/ubuntu/myenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From /home/ubuntu/myenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1070: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
2019-06-01 10:43:52.281117: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-06-01 10:46:03.020289: W tensorflow/core/util/ctc/ctc_loss_calculator.cc:144] No valid path found."`

Can anyone help me in understanding why im encountering this issue ? Any help would be off great use.

Thanks in advance.

blisc commented 5 years ago

It is normal during the beginning of training due to the construction of ctc loss and the random initialization. If there are multiple warnings during the middle of training, then it is a sign that your model is diverging.

pratapaprasanna commented 5 years ago

ok thanks @blisc

aayushkubb commented 4 years ago

@blisc Is there a way to reduce it? Should we tune parameters or try to reduce our vocab?