search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.5k stars 3.49k forks source link

Changing inference type for language models. #212

Open fminkin opened 7 years ago

fminkin commented 7 years ago

Hello! Is it possible to change inference type for lms from sampling next sentence based on target to calculating perplexity of target sentence as it is without changing too much code? The way I see it is to add tensor with per-sentence cross-entropy to the model estimator (if it's possible, I'm not familiar with tf.learn that much), but it'll require too much of code changing :( Any cheap way of doing that?

lukaszkaiser commented 7 years ago

It is quite ugly to do that with Estimator. But it's easy in the new release (1.2.0) to run with pure Session and placeholders, then you get the perplexities as numpy array and can do anything. See this test for how to do it end-to-end: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/trainer_utils_test.py#L96

fminkin commented 7 years ago

@lukaszkaiser

Thanks, I'm currently trying to launch that test, but it fails. Is there any problem on my side?

`====================================================================== ERROR: testSingleStep (main.TrainerUtilsTest)

Traceback (most recent call last): File "s.py", line 93, in testSingleStep exp.test() File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 620, in test hooks=self._train_monitors) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 665, in _call_train monitors=hooks) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 289, in new_func return func(*args, *kwargs) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 455, in fit loss = self._train_model(input_fn=input_fn, hooks=hooks) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 953, in _train_model features, labels = input_fn() File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 103, in input_fn length_multiplier=(p_hparams.batch_size_multiplier))) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensor2tensor/utils/data_reader.py", line 263, in input_pipeline dataset = read_examples(problem, data_file_pattern, capacity, mode=mode) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensor2tensor/utils/data_reader.py", line 236, in read_examples data_items_to_decoders=data_items_to_decoders) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensor2tensor/utils/data_reader.py", line 122, in examples_reader dataset = dataset.map(decode_record, num_threads=num_threads) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 813, in map return MapDataset(self, map_func, num_threads, output_buffer_size) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 1436, in init self._map_func.add_to_graph(ops.get_default_graph()) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 619, in add_to_graph self._create_definition_if_needed() File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/data/python/framework/function.py", line 167, in _create_definition_if_needed outputs = self._func(inputs) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 1427, in tf_map_func flattened_ret = [ops.convert_to_tensor(t) for t in nest.flatten(ret)] File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 676, in convert_to_tensor as_ref=False) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 741, in internal_convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 113, in _constant_tensor_conversion_function return constant(v, dtype=dtype, name=name) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 102, in constant tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape)) File "/home/f-minkin/.pyenv/versions/tflow/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 462, in make_tensor_proto "supported type." % (type(values), values)) TypeError: Failed to convert object of type <type 'dict'> to Tensor. Contents: {'inputs': <tf.Tensor 'SparseToDense:0' shape=(?,) dtype=int64>, 'targets': <tf.Tensor 'SparseToDense_1:0' shape=(?,) dtype=int64>}. Consider casting elements to a supported type.

Ran 5 tests in 6.353s

FAILED (errors=1)`

fminkin commented 7 years ago

@lukaszkaiser Also, can you explain, why does code above for raw session, given model_name = "attention_lm", return probabilties for each word with 794k vocabulary? Can I get probabilities of subwords instead?

rsepassi commented 7 years ago

Looks like you need to upgrade to TensorFlow 1.3.