afrozenator
commented
6 years ago Thanks! Do you want to send a PR for this?
On Thu, Nov 1, 2018, 7:49 AM Michael Mezher <notifications@github.com wrote:
Description
Weights are unable to initialize for the symbol modality due to reuse being set to true in line 146 of modalities.py. Setting reuse to tf.AUTO_REUSE allows the weights to initialize which solves the problem. I see there are other locations within modalities.py where reuse is set to True rather than AUTO_REUSE as well.
Trace: File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2819, in _train_on_tpu_system device_assignment=ctx.device_assignment) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu.py", line 851, in shard name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu.py", line 508, in replicate device_assignment, name)[1] File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu.py", line 659, in split_compile_and_replicate outputs = computation(*computation_inputs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2812, in multi_tpu_train_steps_on_single_shard single_tpu_train_step, [_INITIAL_LOSS]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 207, in repeat cond, body_wrapper, inputs=inputs, infeed_queue=infeed_queue, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 169, in while_loop name="") File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3274, in while_loop return_same_structure) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2994, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2929, in _BuildLoop body_result = body(
packed_vars_for_body) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 120, in body_wrapper outputs = body((inputs + dequeue_ops)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 203, in body_wrapper return [i + 1] + _convert_to_list(body(*args)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 1286, in train_step self._call_model_fn(features, labels)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 1524, in _call_model_fn estimator_spec = self._model_fn(features=features, kwargs) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1251, in wrapping_model_fn use_tpu=use_tpu) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1312, in estimator_model_fn logits, losses_dict = model(features) # pylint: disable=not-callable File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 364, in call outputs = super(Layer, self).call(inputs, *args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 769, in call outputs = self.call(inputs, args, kwargs) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 218, in call sharded_logits, losses = self.model_fn_sharded(sharded_features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 279, in model_fn_sharded sharded_logits, sharded_losses = dp(self.model_fn, datashard_to_features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py", line 231, in call outputs.append(fns[i](*my_args[i],
my_kwargs[i])) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 323, in model_fn logits = self.top(output, features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 497, in top return self._top_single(body_output, target_modality, features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 426, in _top_single logits = target_modality.top(body_output, features.get("targets")) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py", line 154, in top var = self._get_weights(body_output_shape[-1]) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py", line 90, in _get_weights initializer=tf.random_normal_initializer(0.0, hidden_dim-0.5))) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1484, in get_variable aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1234, in get_variable aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 538, in get_variable aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 492, in _true_getter aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 877, in _get_single_variable "reuse=tf.AUTO_REUSE in VarScope?" % name) ValueError: Variable transformer/symbol_modality_40120_512/shared/weights_0 does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope? Environment information
tf = 1.11 t2t = 1.10.0
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensor2tensor/issues/1195, or mute the thread https://github.com/notifications/unsubscribe-auth/ABFCFloKwA2kPkBslWthqhJ47B2Ogy9xks5uqwnfgaJpZM4YGgrb .
mikeymezher
Description
Weights are unable to initialize for the symbol modality due to reuse being set to true in line 146 of modalities.py. Setting reuse to tf.AUTO_REUSE allows the weights to initialize which solves the problem. I see there are other locations within modalities.py where reuse is set to True rather than AUTO_REUSE as well.
Trace: File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2819, in _train_on_tpu_system device_assignment=ctx.device_assignment) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu.py", line 851, in shard name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu.py", line 508, in replicate device_assignment, name)[1] File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu.py", line 659, in split_compile_and_replicate outputs = computation(computation_inputs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2812, in multi_tpu_train_steps_on_single_shard single_tpu_train_step, [_INITIAL_LOSS]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 207, in repeat cond, body_wrapper, inputs=inputs, infeed_queue=infeed_queue, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 169, in while_loop name="") File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3274, in while_loop return_same_structure) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2994, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2929, in _BuildLoop body_result = body(packed_vars_for_body) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 120, in body_wrapper outputs = body((inputs + dequeue_ops)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/training_loop.py", line 203, in body_wrapper return [i + 1] + _convert_to_list(body(args)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 1286, in train_step self._call_model_fn(features, labels)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 1524, in _call_model_fn estimator_spec = self._model_fn(features=features, kwargs) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1251, in wrapping_model_fn use_tpu=use_tpu) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1312, in estimator_model_fn logits, losses_dict = model(features) # pylint: disable=not-callable File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 364, in call outputs = super(Layer, self).call(inputs, *args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 769, in call outputs = self.call(inputs, args, kwargs) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 218, in call sharded_logits, losses = self.model_fn_sharded(sharded_features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 279, in model_fn_sharded sharded_logits, sharded_losses = dp(self.model_fn, datashard_to_features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py", line 231, in call outputs.append(fns[i](*my_args[i], my_kwargs[i])) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 323, in model_fn logits = self.top(output, features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 497, in top return self._top_single(body_output, target_modality, features) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 426, in _top_single logits = target_modality.top(body_output, features.get("targets")) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py", line 154, in top var = self._get_weights(body_output_shape[-1]) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py", line 90, in _get_weights initializer=tf.random_normal_initializer(0.0, hidden_dim-0.5))) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1484, in get_variable aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1234, in get_variable aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 538, in get_variable aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 492, in _true_getter aggregation=aggregation) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 877, in _get_single_variable "reuse=tf.AUTO_REUSE in VarScope?" % name) ValueError: Variable transformer/symbol_modality_40120_512/shared/weights_0 does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?
Environment information
tf = 1.11 t2t = 1.10.0