Kyubyong / tacotron

A TensorFlow Implementation of Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model
Apache License 2.0
1.83k stars 436 forks source link

eval.py error #38

Open sbhamad opened 7 years ago

sbhamad commented 7 years ago

`in ~/tacotron ± |master ?:1 ✗| → python eval.py Graph loaded 2017-06-11 04:22:40.230524: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 04:22:40.230560: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 04:22:40.230565: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 04:22:40.230590: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 04:22:40.230596: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 04:22:40.319841: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2017-06-11 04:22:40.320116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: name: GeForce GTX 850M major: 5 minor: 0 memoryClockRate (GHz) 0.9015 pciBusID 0000:01:00.0 Total memory: 3.95GiB Free memory: 3.60GiB 2017-06-11 04:22:40.320130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 2017-06-11 04:22:40.320134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y 2017-06-11 04:22:40.320153: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 850M, pci bus id: 0000:01:00.0) WARNING:tensorflow:Standard services need a 'logdir' passed to the SessionManager 2017-06-11 04:23:51.726739: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense2/bias not found in checkpoint 2017-06-11 04:23:51.726739: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense1/bias not found in checkpoint 2017-06-11 04:23:51.726877: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.727398: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.728005: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense1/bias not found in checkpoint 2017-06-11 04:23:51.728093: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/beta not found in checkpoint 2017-06-11 04:23:51.728186: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.731562: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense2/bias not found in checkpoint 2017-06-11 04:23:51.731562: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.732466: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/dense/bias not found in checkpoint 2017-06-11 04:23:51.732960: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.733981: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.735061: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.735164: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense2/bias not found in checkpoint 2017-06-11 04:23:51.736522: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_1/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.737246: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.737754: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.737792: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/beta not found in checkpoint 2017-06-11 04:23:51.738832: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.739028: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/dense/kernel not found in checkpoint 2017-06-11 04:23:51.739286: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.739904: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/memory_layer/kernel not found in checkpoint 2017-06-11 04:23:51.740150: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_2/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.741373: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.741403: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/beta not found in checkpoint 2017-06-11 04:23:51.742257: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.742330: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/memory_layer/bias not found in checkpoint 2017-06-11 04:23:51.742398: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense1/bias not found in checkpoint 2017-06-11 04:23:51.742465: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/query_layer/bias not found in checkpoint 2017-06-11 04:23:51.743168: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.748775: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/query_layer/kernel not found in checkpoint 2017-06-11 04:23:51.748925: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/attention_v not found in checkpoint 2017-06-11 04:23:51.750577: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.750620: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.750685: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.751345: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.751419: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.751461: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/beta not found in checkpoint 2017-06-11 04:23:51.752684: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/candidate/biases not found in checkpoint 2017-06-11 04:23:51.756826: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/beta not found in checkpoint 2017-06-11 04:23:51.756855: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.757826: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.758558: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.758617: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.758697: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.759363: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/beta not found in checkpoint 2017-06-11 04:23:51.759437: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.761010: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.762551: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.762665: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.762733: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.763029: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/candidate/biases not found in checkpoint 2017-06-11 04:23:51.763490: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.766689: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.768067: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.768132: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.769356: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.769408: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/beta not found in checkpoint 2017-06-11 04:23:51.770004: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.770070: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/beta not found in checkpoint 2017-06-11 04:23:51.770872: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.771978: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.772033: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.772155: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.773207: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.773297: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.774065: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/candidate/biases not found in checkpoint 2017-06-11 04:23:51.774094: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.774841: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.774933: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense2/bias not found in checkpoint 2017-06-11 04:23:51.777732: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/beta not found in checkpoint 2017-06-11 04:23:51.778343: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.778528: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/dense/bias not found in checkpoint 2017-06-11 04:23:51.778903: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.779435: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.779962: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.781445: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.781533: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/dense/kernel not found in checkpoint 2017-06-11 04:23:51.781463: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.782194: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.782282: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.782316: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.782341: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/beta not found in checkpoint 2017-06-11 04:23:51.783085: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.783132: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.783198: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense1/bias not found in checkpoint 2017-06-11 04:23:51.784151: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.786025: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.786712: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.788681: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.788736: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/beta not found in checkpoint 2017-06-11 04:23:51.788836: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/beta not found in checkpoint 2017-06-11 04:23:51.789438: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.791091: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.793944: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/dense/kernel not found in checkpoint 2017-06-11 04:23:51.794508: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense2/bias not found in checkpoint 2017-06-11 04:23:51.794734: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/dense/bias not found in checkpoint 2017-06-11 04:23:51.795663: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.795714: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.795780: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/candidate/biases not found in checkpoint 2017-06-11 04:23:51.797021: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.797069: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.798374: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.798547: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.799517: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.799608: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.800093: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/candidate/biases not found in checkpoint 2017-06-11 04:23:51.800156: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/beta not found in checkpoint 2017-06-11 04:23:51.801278: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense1/bias not found in checkpoint 2017-06-11 04:23:51.801840: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.801920: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.801947: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.802877: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.802956: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.802997: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.804123: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense1/bias not found in checkpoint 2017-06-11 04:23:51.804394: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.805234: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.805775: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.806277: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.806354: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.806388: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense2/bias not found in checkpoint 2017-06-11 04:23:51.807468: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.807864: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.807918: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense2/bias not found in checkpoint 2017-06-11 04:23:51.807991: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/beta not found in checkpoint 2017-06-11 04:23:51.809839: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense1/bias not found in checkpoint 2017-06-11 04:23:51.810985: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.811027: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/beta not found in checkpoint 2017-06-11 04:23:51.811150: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense2/bias not found in checkpoint 2017-06-11 04:23:51.811526: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.811715: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.812429: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.812795: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.813020: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.813227: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.813480: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/beta not found in checkpoint 2017-06-11 04:23:51.813531: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.814402: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.817649: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.818524: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.818568: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/beta not found in checkpoint 2017-06-11 04:23:51.818709: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_2/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.818765: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense1/bias not found in checkpoint 2017-06-11 04:23:51.819173: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.819656: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.819738: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.821486: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_1/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.821552: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.822615: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.823723: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.824655: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.824797: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.825653: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense2/kernel not found in checkpoint 2017-06-11 04:23:51.825751: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.826833: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense2/bias not found in checkpoint 2017-06-11 04:23:51.827897: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/beta not found in checkpoint 2017-06-11 04:23:51.828765: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.828771: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense2/bias not found in checkpoint 2017-06-11 04:23:51.828834: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/beta not found in checkpoint 2017-06-11 04:23:51.828888: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/beta not found in checkpoint 2017-06-11 04:23:51.829065: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.830252: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.831061: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.831519: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.831819: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.834723: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense1/kernel not found in checkpoint 2017-06-11 04:23:51.834823: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.835854: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.836123: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.836597: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/beta not found in checkpoint 2017-06-11 04:23:51.836692: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.836778: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense1/bias not found in checkpoint 2017-06-11 04:23:51.837524: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.838518: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.838617: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.839402: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense1/bias not found in checkpoint 2017-06-11 04:23:51.839452: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.840713: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/beta not found in checkpoint 2017-06-11 04:23:51.840798: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.840875: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.841344: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.841399: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/beta not found in checkpoint 2017-06-11 04:23:51.841449: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.843741: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.843952: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.844243: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.845037: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.845137: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.845398: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.846067: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/beta not found in checkpoint 2017-06-11 04:23:51.846391: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.846549: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.846789: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.847836: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.847907: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.848012: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.849559: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/beta not found in checkpoint 2017-06-11 04:23:51.849627: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.849656: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.852251: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/moving_mean not found in checkpoint 2017-06-11 04:23:51.852984: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/conv1d/conv1d/kernel not found in checkpoint 2017-06-11 04:23:51.853197: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.854297: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/gamma not found in checkpoint 2017-06-11 04:23:51.854955: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/beta not found in checkpoint 2017-06-11 04:23:51.855167: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/gates/biases not found in checkpoint 2017-06-11 04:23:51.855385: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/gates/weights not found in checkpoint 2017-06-11 04:23:51.855455: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/embedding/lookup_table not found in checkpoint 2017-06-11 04:23:51.855525: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/moving_variance not found in checkpoint 2017-06-11 04:23:51.855979: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/candidate/biases not found in checkpoint 2017-06-11 04:23:51.856067: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/candidate/weights not found in checkpoint 2017-06-11 04:23:51.856384: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/candidate/biases not found in checkpoint Traceback (most recent call last): File "eval.py", line 71, in eval() File "eval.py", line 38, in eval sv.saver.restore(sess, tf.train.latest_checkpoint(hp.logdir)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1457, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 778, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 982, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: Key decoder2/highwaynet_2/dense2/bias not found in checkpoint [[Node: save/RestoreV2_87 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_87/tensor_names, save/RestoreV2_87/shape_and_slices)]] [[Node: save/RestoreV2_22/_631 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1470_save/RestoreV2_22", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_87', defined at: File "eval.py", line 71, in eval() File "eval.py", line 35, in eval sv = tf.train.Supervisor() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 300, in init self._init_saver(saver=saver) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 446, in _init_saver saver = saver_mod.Saver() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1056, in init self.build() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1086, in build restore_sequentially=self._restore_sequentially) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 691, in build restore_sequentially, reshape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps tensors = self.restore_op(filename_tensor, saveable, preferred_shard) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 247, in restore_op [spec.tensor.dtype])[0]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2 dtypes=dtypes, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1228, in init self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key decoder2/highwaynet_2/dense2/bias not found in checkpoint [[Node: save/RestoreV2_87 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_87/tensor_names, save/RestoreV2_87/shape_and_slices)]] [[Node: save/RestoreV2_22/_631 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1470_save/RestoreV2_22", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

`

msobhan69 commented 7 years ago

I had this problem, this is for length of sound in evaluation batch (max_duration=5.0s in hyperparameters.py)

sbhamad commented 7 years ago

i think the problem lies within eval.py, pylint is red underlining many expressions in the code:

this is the eval.py file content:

` from future import print_function import codecs import copy import os

import tensorflow as tf import numpy as np

import librosa from scipy.io.wavfile import write

from hyperparams import Hyperparams as hp from prepro import if hp.num_gpus == 1: from train import Graph else: from train_multi_gpus import Graph from utils import

def eval():

Load graph

g = Graph(is_training=False)
print("Graph loaded")

# Load data
X = load_eval_data() # texts
char2idx, idx2char = load_vocab()

with g.graph.as_default():    
    sv = tf.train.Supervisor()
    with sv.managed_session(config=tf.ConfigProto(allow_soft_placement=True)) as sess:
        # Restore parameters
        sv.saver.restore(sess, tf.train.latest_checkpoint(hp.logdir))
        print("Restored!")

        # Get model
        mname = open(hp.logdir + '/checkpoint', 'r').read().split('"')[1] # model name

        timesteps = 100 # Adjust this number as you want
        outputs_shifted = np.zeros((hp.batch_size, timesteps, hp.n_mels*hp.r), np.int32)
        outputs = np.zeros((hp.batch_size, timesteps, hp.n_mels*hp.r), np.float32)   # hp.n_mels*hp.r  
        for j in range(timesteps):
            # predict next frames
            _outputs = sess.run(g.outputs1, {g.x: X, g.decoder_inputs: outputs_shifted})
            # update character sequence
            if j < timesteps - 1:
                outputs_shifted[:, j + 1] = _outputs[:, j, :]
            outputs[:, j, :] = _outputs[:, j, :]

        outputs2 = sess.run(g.outputs2, {g.outputs1: outputs})
        spectrograms = restore_shape(outputs2, hp.r)

# Generate wav files
if not os.path.exists('samples'): os.mkdir('samples') 
with codecs.open('samples/text.txt', 'w', 'utf-8') as fout:
    for i, (x, s) in enumerate(zip(X, spectrograms)):
        # write text
        fout.write(str(i) + "\t" + "".join(idx2char[idx] for idx in np.fromstring(x, np.int32) if idx != 0) + "\n")

        # generate wav files
        audio = spectrogram2wav(np.power(np.e, s)**hp.power)
        write("samples/{}_{}.wav".format(mname, i), hp.sr, audio)     

if name == 'main': eval() print("Done") `

hmubarak commented 7 years ago

I have a similar problem! Training on a GPU machine for 2 days for the bible data you share was done successfully, but calling eval.py didn't produce any wave files, and gave the following: python eval.py Graph loaded 2017-06-15 08:27:21.395031: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-15 08:27:21.395185: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-15 08:27:21.395220: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. WARNING:tensorflow:Standard services need a 'logdir' passed to the SessionManager 2017-06-15 08:27:21.726531: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.726561: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.726586: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.726635: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.726792: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/beta not found in checkpoint 2017-06-15 08:27:21.726841: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.727044: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/normalize/beta not found in checkpoint 2017-06-15 08:27:21.727350: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.727680: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/beta not found in checkpoint 2017-06-15 08:27:21.727712: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.727985: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.728544: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.728588: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense1/bias not found in checkpoint 2017-06-15 08:27:21.728760: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.728761: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_12/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.729010: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.729539: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense1/bias not found in checkpoint 2017-06-15 08:27:21.729632: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.729781: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense2/bias not found in checkpoint 2017-06-15 08:27:21.729908: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_13/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.730085: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.730149: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense2/bias not found in checkpoint 2017-06-15 08:27:21.730333: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_2/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.730663: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.730756: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.730866: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.731375: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/beta not found in checkpoint 2017-06-15 08:27:21.731494: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_1/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.731538: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_2/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.731670: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_3/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.731917: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.732214: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.732443: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.732573: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.732781: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/beta not found in checkpoint 2017-06-15 08:27:21.733163: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru1/rnn/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.733245: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.733383: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.734123: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/dense/bias not found in checkpoint 2017-06-15 08:27:21.734186: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.734272: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.734311: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.734321: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.734368: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.734550: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.735037: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/dense/kernel not found in checkpoint 2017-06-15 08:27:21.735099: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.735294: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.736003: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.736071: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.736161: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.736278: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/fw/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.736322: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense1/bias not found in checkpoint 2017-06-15 08:27:21.736344: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/gru/bidirectional_rnn/bw/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.738537: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense1/bias not found in checkpoint 2017-06-15 08:27:21.739096: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.739580: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.740018: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.740449: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.741020: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/beta not found in checkpoint 2017-06-15 08:27:21.741888: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.742137: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense2/bias not found in checkpoint 2017-06-15 08:27:21.742349: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.742385: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_0/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.742489: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.742535: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/embedding/lookup_table not found in checkpoint 2017-06-15 08:27:21.742718: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.742904: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.743253: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.743425: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/beta not found in checkpoint 2017-06-15 08:27:21.743480: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.743660: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_3/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.743871: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.743936: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.744191: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.744558: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_4/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.744630: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/beta not found in checkpoint 2017-06-15 08:27:21.744977: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.745290: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.745467: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.745578: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.745754: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.745795: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_5/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.745851: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/beta not found in checkpoint 2017-06-15 08:27:21.746016: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.746229: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.746968: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_6/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.747009: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.747055: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/beta not found in checkpoint 2017-06-15 08:27:21.747322: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.747354: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.747481: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_14/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.747789: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/rnn/attention_v not found in checkpoint 2017-06-15 08:27:21.748092: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.748379: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.751770: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.751978: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.752038: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.752076: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.752224: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_15/normalize/beta not found in checkpoint 2017-06-15 08:27:21.752345: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/beta not found in checkpoint 2017-06-15 08:27:21.752507: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.753272: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.753480: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.753550: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.753618: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.753866: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.754119: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/beta not found in checkpoint 2017-06-15 08:27:21.754240: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_16/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.754571: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.754640: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_2/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.754732: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/beta not found in checkpoint 2017-06-15 08:27:21.755118: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/beta not found in checkpoint 2017-06-15 08:27:21.755350: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_11/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.755406: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/bw/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.755471: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.755834: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_10/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.756083: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.756259: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/candidate/biases not found in checkpoint 2017-06-15 08:27:21.756309: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.756633: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.757151: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/gru/bidirectional_rnn/fw/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.757192: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.757319: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense1/bias not found in checkpoint 2017-06-15 08:27:21.757400: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense1/bias not found in checkpoint 2017-06-15 08:27:21.757494: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense2/bias not found in checkpoint 2017-06-15 08:27:21.757955: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.757992: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.758273: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense1/bias not found in checkpoint 2017-06-15 08:27:21.758371: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.758433: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_0/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.758552: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.758793: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.759407: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_1/dense2/bias not found in checkpoint 2017-06-15 08:27:21.759711: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.759730: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense2/bias not found in checkpoint 2017-06-15 08:27:21.760008: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.760109: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_2/dense2/bias not found in checkpoint 2017-06-15 08:27:21.760198: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense1/bias not found in checkpoint 2017-06-15 08:27:21.760371: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.760746: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.761038: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.761089: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/beta not found in checkpoint 2017-06-15 08:27:21.761313: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.761337: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.761508: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_7/normalize/beta not found in checkpoint 2017-06-15 08:27:21.762481: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.762931: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/beta not found in checkpoint 2017-06-15 08:27:21.763384: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.763832: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.764295: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.764791: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense1/bias not found in checkpoint 2017-06-15 08:27:21.765051: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.765463: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.765645: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.765693: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.765748: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/highwaynet_3/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.765795: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_8/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.765847: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.765933: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/conv1d_banks/num_9/normalize/beta not found in checkpoint 2017-06-15 08:27:21.766429: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/normalize/beta not found in checkpoint 2017-06-15 08:27:21.767134: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/gates/weights not found in checkpoint 2017-06-15 08:27:21.767302: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/candidate/weights not found in checkpoint 2017-06-15 08:27:21.767362: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense2/bias not found in checkpoint 2017-06-15 08:27:21.767394: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/decoder_gru2/rnn/gru_cell/gates/biases not found in checkpoint 2017-06-15 08:27:21.767457: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/dense/kernel not found in checkpoint 2017-06-15 08:27:21.767555: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key encoder/prenet/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.767826: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/dense/bias not found in checkpoint 2017-06-15 08:27:21.767969: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_3/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.768416: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.768510: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense2/bias not found in checkpoint 2017-06-15 08:27:21.768668: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_1/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.768952: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/dense/bias not found in checkpoint 2017-06-15 08:27:21.769369: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense1/kernel not found in checkpoint 2017-06-15 08:27:21.769629: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/dense/kernel not found in checkpoint 2017-06-15 08:27:21.769703: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/prenet/dense1/bias not found in checkpoint 2017-06-15 08:27:21.769777: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/memory_layer/kernel not found in checkpoint 2017-06-15 08:27:21.769859: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/memory_layer/bias not found in checkpoint 2017-06-15 08:27:21.770335: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/query_layer/bias not found in checkpoint 2017-06-15 08:27:21.770529: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.770652: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_4/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.770688: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder1/attention_decoder/query_layer/kernel not found in checkpoint 2017-06-15 08:27:21.770877: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.771323: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.771564: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense2/kernel not found in checkpoint 2017-06-15 08:27:21.771801: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.771993: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/beta not found in checkpoint 2017-06-15 08:27:21.772032: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/highwaynet_1/dense2/bias not found in checkpoint 2017-06-15 08:27:21.772094: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.772329: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_5/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.772504: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/beta not found in checkpoint 2017-06-15 08:27:21.772892: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.772955: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.773071: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.773148: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_6/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.773278: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.773730: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/beta not found in checkpoint 2017-06-15 08:27:21.773780: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.773840: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/moving_mean not found in checkpoint 2017-06-15 08:27:21.773915: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.773975: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_7/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.774860: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.774890: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_2/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.774979: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.775161: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.775230: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_8/normalize/beta not found in checkpoint 2017-06-15 08:27:21.775356: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/beta not found in checkpoint 2017-06-15 08:27:21.775720: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/moving_variance not found in checkpoint 2017-06-15 08:27:21.775770: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/conv1d/conv1d/kernel not found in checkpoint 2017-06-15 08:27:21.775822: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/num_2/normalize/beta not found in checkpoint 2017-06-15 08:27:21.775889: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/gamma not found in checkpoint 2017-06-15 08:27:21.775915: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key decoder2/conv1d_banks/normalize/moving_mean not found in checkpoint Traceback (most recent call last): File "eval.py", line 71, in eval() File "eval.py", line 38, in eval sv.saver.restore(sess, tf.train.latest_checkpoint(hp.logdir)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1457, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 778, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 982, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: Key encoder/conv1d_banks/num_11/normalize/moving_variance not found in checkpoint [[Node: save/RestoreV2_113 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_113/tensor_names, save/RestoreV2_113/shape_and_slices)]]

Caused by op u'save/RestoreV2_113', defined at: File "eval.py", line 71, in eval() File "eval.py", line 35, in eval sv = tf.train.Supervisor() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 300, in init self._init_saver(saver=saver) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 446, in _init_saver saver = saver_mod.Saver() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1056, in init self.build() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1086, in build restore_sequentially=self._restore_sequentially) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 691, in build restore_sequentially, reshape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps tensors = self.restore_op(filename_tensor, saveable, preferred_shard) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 247, in restore_op [spec.tensor.dtype])[0]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2 dtypes=dtypes, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1228, in init self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key encoder/conv1d_banks/num_11/normalize/moving_variance not found in checkpoint [[Node: save/RestoreV2_113 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_113/tensor_names, save/RestoreV2_113/shape_and_slices)]]

danijel3 commented 7 years ago

Had the same error and found a solution. If you train using train_multi_gpu, make sure you change the gpu count in the hyperparameters file. Eval will use this option to determine which graph (single or multi) to use.

aelbialy-tbox commented 7 years ago

I got the same error. train_multi_gpu == 2 in my case. I know eval.py got updated in this repo but I am still getting the same error any solutions guys? Here's my stack `Graph loaded 2017-07-18 16:59:47.132451: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-07-18 16:59:47.132510: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-07-18 16:59:47.132519: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2017-07-18 16:59:47.132530: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2017-07-18 16:59:47.132537: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2017-07-18 16:59:50.850896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2017-07-18 16:59:50.851298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: name: Tesla M60 major: 5 minor: 2 memoryClockRate (GHz) 1.1775 pciBusID 0000:00:1d.0 Total memory: 7.43GiB Free memory: 7.36GiB 2017-07-18 16:59:50.909166: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x5a69cd0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that. 2017-07-18 16:59:50.909568: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2017-07-18 16:59:50.909952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 1 with properties: name: Tesla M60 major: 5 minor: 2 memoryClockRate (GHz) 1.1775 pciBusID 0000:00:1e.0 Total memory: 7.43GiB Free memory: 7.36GiB 2017-07-18 16:59:50.910011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:832] Peer access not supported between device ordinals 0 and 1 2017-07-18 16:59:50.910035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:832] Peer access not supported between device ordinals 1 and 0 2017-07-18 16:59:50.910062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0 1 2017-07-18 16:59:50.910074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: Y N 2017-07-18 16:59:50.910079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 1: N Y 2017-07-18 16:59:50.910090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla M60, pci bus id: 0000:00:1d.0) 2017-07-18 16:59:50.910097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla M60, pci bus id: 0000:00:1e.0) WARNING:tensorflow:Standard services need a 'logdir' passed to the SessionManager 2017-07-18 16:59:51.540125: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint 2017-07-18 16:59:51.547393: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable not found in checkpoint 2017-07-18 16:59:51.563246: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.565312: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.566152: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.568060: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.570097: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.572284: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.573463: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.573659: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.575815: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.576565: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.578894: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.580359: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.582498: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.582801: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.583087: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.583172: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm1/Variable_1 not found in checkpoint 2017-07-18 16:59:51.583514: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.583897: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.585309: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.585475: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.586031: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.586067: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.586283: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.588576: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.589434: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.590058: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.591135: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.591292: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.593825: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.595552: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.595786: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.595977: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.598858: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.598992: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.599005: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm1/Variable not found in checkpoint 2017-07-18 16:59:51.601067: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.601143: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.601720: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.602208: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.602350: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.603454: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.604535: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.606954: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.607091: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.607566: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.608603: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.610012: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.610448: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.610689: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.612287: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.612310: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.612644: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.614259: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.614439: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.618568: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.620911: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.621612: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.621995: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.623787: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.624080: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.624532: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.625002: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.626727: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.627053: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.627228: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.630176: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/decoder2/conv1d_banks/normalize/Variable not found in checkpoint 2017-07-18 16:59:51.631228: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.633243: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.633300: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.633936: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.635118: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.635460: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.636113: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.638336: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/decoder2/conv1d_banks/normalize/Variable_1 not found in checkpoint 2017-07-18 16:59:51.642223: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.644253: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.644579: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.648063: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.651817: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.653302: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.653490: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.653680: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.659624: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.660426: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.660497: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.661359: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.661793: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.661902: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.661956: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.662373: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.662763: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.663474: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.666532: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.668314: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.668368: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.668553: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/decoder2/norm1/Variable_1 not found in checkpoint 2017-07-18 16:59:51.669244: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.669298: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.669470: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.670692: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/decoder2/norm2/Variable_1 not found in checkpoint 2017-07-18 16:59:51.670942: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.671824: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.673393: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.674538: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.674636: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/conv1d_banks/normalize/Variable_1 not found in checkpoint 2017-07-18 16:59:51.675523: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/decoder2/norm2/Variable not found in checkpoint 2017-07-18 16:59:51.675945: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.676118: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.676448: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.677067: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/decoder2/norm1/Variable not found in checkpoint 2017-07-18 16:59:51.677226: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.677486: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.677651: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.677976: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.678667: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.680008: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/conv1d_banks/normalize/Variable not found in checkpoint 2017-07-18 16:59:51.680064: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.680264: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.680346: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] 2017-07-18 16:59:51.680443: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1139, in _do_call return fn(*args) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1121, in _run_fn status, run_metadata) File "/usr/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.NotFoundError: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] [[Node: save/RestoreV2_99/_445 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_434_save/RestoreV2_99", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "eval.py", line 70, in eval() File "eval.py", line 39, in eval sv.saver.restore(sess, tf.train.latest_checkpoint(hp.logdir)) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1548, in restore {self.saver_def.filename_tensor_name: save_path}) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 789, in run run_metadata_ptr) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 997, in _run feed_dict_string, options, run_metadata) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run target_list, options, run_metadata) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] [[Node: save/RestoreV2_99/_445 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_434_save/RestoreV2_99", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op 'save/RestoreV2_118', defined at: File "eval.py", line 70, in eval() File "eval.py", line 36, in eval sv = tf.train.Supervisor() File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/supervisor.py", line 300, in init self._init_saver(saver=saver) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/supervisor.py", line 448, in _init_saver saver = saver_mod.Saver() File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1139, in init self.build() File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1170, in build restore_sequentially=self._restore_sequentially) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 691, in build restore_sequentially, reshape) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps tensors = self.restore_op(filename_tensor, saveable, preferred_shard) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op [spec.tensor.dtype])[0]) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 640, in restore_v2 dtypes=dtypes, name=name) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1269, in init self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key net/encoder/norm2/Variable_1 not found in checkpoint [[Node: save/RestoreV2_118 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_118/tensor_names, save/RestoreV2_118/shape_and_slices)]] [[Node: save/RestoreV2_99/_445 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_434_save/RestoreV2_99", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

`

jpdz commented 7 years ago

@ahmed-tbox Have you solved this problem? Same problem with you.

qclu commented 7 years ago

I have tried mult-gpu train on bible data, and got same errors. Still I tried with train.py, I also got same errors. @danijel3, would u pls explain your solution in detail?

danijel3 commented 7 years ago

My comment was referring to an older version of the project. I tried with the current version and I couldn't fix it.

A couple of fixes I managed to figure out:

Both these fixes didn't help in my case, but maybe you'll have more luck...