search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.5k stars 3.49k forks source link

[bug] ValueError in t2t-decoder during text_cnn model initialization for Text2Class problem #1562

Open bnjmacdonald opened 5 years ago

bnjmacdonald commented 5 years ago

Description

I have trained a text_cnn model on a custom Text2Class problem without any issues using t2t-trainer (I've also continued training with t2t-trainer by restoring from a checkpoint without any issues), but when I use t2t-decoder I get the ValueError below.

I have not had any issues in training or decoding when using the exact same code but with different models (e.g. transformer_encoder, lstm_encoder).

Environment information

OS: Mac OS 10.14.4

$ pip freeze | grep tensor
tensor2tensor==1.13.2
tensorboard==1.13.1
tensorflow==1.13.1
tensorflow-datasets==1.0.1
tensorflow-estimator==1.13.0
tensorflow-hub==0.4.0
tensorflow-metadata==0.13.0
tensorflow-probability==0.6.0

$ python -V
Python 3.6.5

For bugs: reproduction and error logs

# Error logs:
> PROBLEM="hansard_line_type4"  # custom problem, subclass of Text2Class problem.
> MODEL="text_cnn"
> HPARAMS_SET="text_cnn_base"
> HPARAMS="batch_size=2048,max_length=2048,num_hidden_layers=3,num_filters=64,hidden_size=64"
> t2t-decoder \
>     --t2t_usr_dir=$USR_DIR \
>     --data_dir=$DATA_DIR \
>     --problem=$PROBLEM \
>     --model=$MODEL \
>     --hparams_set=$HPARAMS_SET \
>     --decode_hparams=$DECODE_HPARAMS \
>     --hparams=$HPARAMS \
>     --output_dir=$TRAIN_DIR \
>     --decode_from_file="$DECODE_FILE" \
>     --decode_to_file=$DECODE_OUTPUT_FILE

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

INFO:tensorflow:Importing user module t2t_problem from path /Users/bmacwell/Documents/current/projects/hansardparser/hansardparser/plenaryparser/TxtParser/SpeakerParser
INFO:tensorflow:Overriding hparams in text_cnn_base with batch_size=2048,max_length=2048,num_hidden_layers=6,num_filters=64
WARNING:tensorflow:From /Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/trainer_lib.py:240: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
INFO:tensorflow:Configuring DataParallelism to replicate the model.
INFO:tensorflow:schedule=continuous_train_and_eval
INFO:tensorflow:worker_gpu=1
INFO:tensorflow:sync=False
WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:ps_devices: ['gpu:0']
INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1443f6b38>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_device_fn': None, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
    global_jit_level: OFF
  }
}
isolate_session_state: true
, '_save_checkpoints_steps': 1000, '_keep_checkpoint_max': 20, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'experiments/plenaryparser/t2t_train/hansard_line_has_speaker/text_cnn/text_cnn_base/experiment0', 'use_tpu': False, 't2t_device_info': {'num_async_replicas': 1}, 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x1443f6ba8>}
WARNING:tensorflow:Estimator's model_fn (<function T2TModel.make_estimator_model_fn.<locals>.wrapping_model_fn at 0x142c2d378>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Performing decoding from file (data/tests/raw/speaker_names.txt).
INFO:tensorflow:Getting sorted inputs
WARNING:tensorflow:From /Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
INFO:tensorflow: batch 5
INFO:tensorflow:Decoding batch 0
WARNING:tensorflow:From /Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/decoding.py:616: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
    tf.py_function, which takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.

WARNING:tensorflow:From /Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/decoding.py:949: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:Input graph does not use tf.data.Dataset or contain a QueueRunner. That means predict yields forever. This is probably a mistake.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights.
INFO:tensorflow:Setting T2TModel mode to 'infer'
INFO:tensorflow:Setting hparams.dropout to 0.0
INFO:tensorflow:Setting hparams.label_smoothing to 0.0
INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0
INFO:tensorflow:Setting hparams.symbol_dropout to 0.0
INFO:tensorflow:Setting hparams.output_dropout to 0.0
INFO:tensorflow:Greedy Decoding
INFO:tensorflow:Using variable initializer: uniform_unit_scaling
WARNING:tensorflow:Missing feature context - ignoring.
INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_258_64.bottom
WARNING:tensorflow:From /Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/framework/function.py:1007: calling Graph.create_op (from tensorflow.python.framework.ops) with compute_shapes is deprecated and will be removed in a future version.
Instructions for updating:
Shapes are always computed; don't use the compute_shapes as it has no effect.
INFO:tensorflow:Transforming feature 'targets' with class_label_modality_2_64.targets_bottom
INFO:tensorflow:Building model body
Traceback (most recent call last):
  File "/Users/bmacwell/envs/hansardparser/bin/t2t-decoder", line 17, in <module>
    tf.app.run()
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/Users/bmacwell/envs/hansardparser/bin/t2t-decoder", line 12, in main
    t2t_decoder.main(argv)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/bin/t2t_decoder.py", line 193, in main
    decode(estimator, hp, decode_hp)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/bin/t2t_decoder.py", line 92, in decode
    checkpoint_path=FLAGS.checkpoint_path)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/decoding.py", line 473, in decode_from_file
    for elapsed_time, result in timer(result_iter):
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/decoding.py", line 467, in timer
    item = next(gen)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 611, in predict
    features, None, model_fn_lib.ModeKeys.PREDICT, self.config)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1112, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 1405, in wrapping_model_fn
    use_tpu=use_tpu)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 1462, in estimator_model_fn
    return model.estimator_spec_predict(features, use_tpu=use_tpu)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 1683, in estimator_spec_predict
    use_tpu=use_tpu)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 803, in infer
    results = self._greedy_infer(features, decode_length, use_tpu)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 967, in _greedy_infer
    return self._slow_greedy_infer(features, decode_length)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 1299, in _slow_greedy_infer
    parallel_iterations=1)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop
    return_same_structure)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 1191, in infer_step
    samples, logits, losses = self.sample(features)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 1329, in sample
    logits, losses = self(features)  # pylint: disable=not-callable
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 530, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 322, in call
    sharded_logits, losses = self.model_fn_sharded(sharded_features)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 383, in model_fn_sharded
    sharded_logits, sharded_losses = dp(self.model_fn, datashard_to_features)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/expert_utils.py", line 231, in __call__
    outputs.append(fns[i](*my_args[i], **my_kwargs[i]))
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/utils/t2t_model.py", line 419, in model_fn
    body_out = self.body(transformed_features)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensor2tensor/models/text_cnn.py", line 59, in body
    tf.truncated_normal(filter_shape, stddev=0.1), name="W")
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 213, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 176, in _variable_v1_call
    aggregation=aggregation)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 155, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2488, in default_variable_creator
    import_scope=import_scope)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 217, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 294, in __init__
    constraint=constraint)
  File "/Users/bmacwell/envs/hansardparser/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 422, in _init_from_args
    "initializer." % name)
ValueError: Initializer for variable while/text_cnn/parallel_0_5/text_cnn/text_cnn/body/conv-maxpool-2/W/ is from inside a control-flow construct, such as a loop or conditional. When creating a variable inside a loop or conditional, use a lambda as the initializer.
Pogben23 commented 5 years ago

Same problem from me as well. Any ideas?

jiahuigeng commented 5 years ago

Anyone solved this probelm?