LearnedVector
commented
6 years ago This seems to be a problem with mismatched tensor dimensions between the checkpoint model and the model at its current state, possibly from different parameters used for the model dimensions. Don't have time to look into it deeply right now but can take a look when I have more bandwidth.
el-tocino
Followed the steps on the readme to try running the demo server. Ubuntu 16.04, python3.6, tf 1.9, cuda 9.0.
Edited the models/tacotron.py and added: "from .attention import BahdanauAttention" and then in the Attention bit, commented out the LocationSensitiveAttention and added "BahdanauAttention(256, encoder_outputs)," (also tried 512 after getting errors, similar problem still occurred) Then I tried running the demo server:
Instructions for updating: seq_dim is deprecated, use seq_axis instead WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:432: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version. Instructions for updating: batch_dim is deprecated, use batch_axis instead Initialized Tacotron model. Dimensions: embedding: 512 prenet out: 128 encoder out: 256 attention out: 256 concat attn & out: 512 decoder cell out: 256 decoder out (5 frames): 400 decoder out (1 frame): 80 postnet out: 256 linear out: 1025 Loading checkpoint: /tmp/tacotron-20170720/model.ckpt Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call return fn(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [512,256] rhs shape= [256,256] [[Node: save/Assign_245 = Assign[T=DT_FLOAT, _class=["loc:@model/inference/prenet/dense_1/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/inference/prenet/dense_1/kernel, save/RestoreV2/_491)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "demo_server.py", line 120, in
synthesizer.load(args.checkpoint)
File "/home/el-tocino/mimic2/synthesizer.py", line 32, in load
saver.restore(self.session, checkpoint_path)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1752, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [512,256] rhs shape= [256,256]
[[Node: save/Assign_245 = Assign[T=DT_FLOAT, _class=["loc:@model/inference/prenet/dense_1/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/inference/prenet/dense_1/kernel, save/RestoreV2/_491)]]
Caused by op 'save/Assign_245', defined at: File "demo_server.py", line 120, in
synthesizer.load(args.checkpoint)
File "/home/el-tocino/mimic2/synthesizer.py", line 31, in load
saver = tf.train.Saver()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1284, in init
self.build()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1296, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1333, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
restore_sequentially, reshape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 113, in restore
self.op.get_shape().is_fully_defined())
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
validate_shape=validate_shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [512,256] rhs shape= [256,256] [[Node: save/Assign_245 = Assign[T=DT_FLOAT, _class=["loc:@model/inference/prenet/dense_1/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/inference/prenet/dense_1/kernel, save/RestoreV2/_491)]]