*bug* can't export model _packed problems

mehmedes commented 6 years ago

Description

t2t-exporter won't export models trained as _packed.

TensorFlow and tensor2tensor versions

tensor2tensor==1.5.5 tensorboard==1.5.0 tensorflow==1.5.0 tensorflow-gpu==1.5.0 tensorflow-serving-api==1.5.0 tensorflow-tensorboard==1.5.0

In case of bug report: Steps to reproduce the problem

t2t-exporter \
  --model=transformer \
  --hparams_set=transformer_big_adafactor \
  --problems=translate_ende_wmt32k_packed \
  --data_dir=/home/t2t_data \
  --output_dir=/home/t2t_train/translate_ende_wmt32k_packed/transformer-tr$

In case of bug report: Error log

Traceback (most recent call last):
  File "export.py", line 75, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "export.py", line 71, in main
    checkpoint_path=tf.train.latest_checkpoint(ckpt_dir))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/export_strategy.py", line 98, in export
    return self.export_fn(estimator, export_path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/utils/saved_model_export_utils.py", line 455, in export_fn
    strip_default_attrs=strip_default_attrs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 592, in export_savedmodel
    config=self.config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 793, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 874, in wrapping_model_fn
    use_tpu=use_tpu)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 919, in estimator_model_fn
    return model.estimator_spec_predict(features)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1037, in estimator_spec_predict
    decode_length=decode_hparams.extra_length)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 523, in infer
    top_beams, alpha)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 230, in _beam_decode
    features, decode_length, beam_size, top_beams, alpha)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 291, in _fast_decode
    features=features)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py", line 254, in __call__
    outputs.append(fns[i](*my_args[i], **my_kwargs[i]))
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 85, in encode
    hparams, nonpadding=features_to_nonpadding(features, "inputs"),
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 536, in features_to_nonpadding
    return tf.minimum(features[key], 1.0)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 2717, in minimum
    "Minimum", x=x, y=y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 519, in _apply_op_helper
    repr(values), type(values).__name__))
TypeError: Expected int32 passed to parameter 'y' of op 'Minimum', got 1.0 of type 'float' instead.

mehmedes commented 6 years ago

FYI: Exporting the model trained on _packed datasets as non-packed --problems=translate_ende_wmt32k works.

mehmedes commented 6 years ago

Thanks, with the latest release _packed probles can be exported!

jurukode commented 6 years ago

Hi @mehmedes,

So what you did actually to solve this issue? is it change the _packed to non-packed or what? Thanks

mehmedes commented 6 years ago

@dyngts : Exporting it and running it as PROBLEM=translate_ende_wmt32k instead of PROBLEM=translate_ende_wmt32k_packed helped me.

tensorflow / tensor2tensor