AttributeError: '_VariantDataset' object has no attribute 'output_shapes'

[loretoparisi]

Hi guys, I have successfully trained a G2P seq2seq model for the IPA dictionary in the first release of this repo. I'm now trying to reproduce the same results with an updated IPA dictionary, and using this new codebase, but I'm getting this error:

Traceback (most recent call last):
  File "/usr/local/bin/g2p-seq2seq", line 11, in <module>
    load_entry_point('g2p-seq2seq==6.2.2a0', 'console_scripts', 'g2p-seq2seq')()
  File "/usr/local/lib/python3.7/site-packages/g2p_seq2seq-6.2.2a0-py3.7.egg/g2p_seq2seq/app.py", line 113, in main
    g2p_model.train()
  File "/usr/local/lib/python3.7/site-packages/g2p_seq2seq-6.2.2a0-py3.7.egg/g2p_seq2seq/g2p.py", line 259, in train
    execute_schedule(self.exp, self.params)
  File "/usr/local/lib/python3.7/site-packages/g2p_seq2seq-6.2.2a0-py3.7.egg/g2p_seq2seq/g2p.py", line 589, in execute_schedule
    getattr(exp, params.schedule)()
  File "/usr/local/lib/python3.7/site-packages/tensor2tensor-1.7.0-py3.7.egg/tensor2tensor/utils/trainer_lib.py", line 309, in continuous_train_and_eval
    self._eval_spec)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1185, in _train_model_default
    input_fn, ModeKeys.TRAIN))
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1022, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1113, in _call_input_fn
    return input_fn(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensor2tensor-1.7.0-py3.7.egg/tensor2tensor/data_generators/problem.py", line 733, in estimator_input_fn
    dataset_kwargs=dataset_kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensor2tensor-1.7.0-py3.7.egg/tensor2tensor/data_generators/problem.py", line 895, in input_fn
    batching_scheme["batch_sizes"])
  File "/usr/local/lib/python3.7/site-packages/tensor2tensor-1.7.0-py3.7.egg/tensor2tensor/utils/data_reader.py", line 103, in bucket_by_sequence_length
    window_size_fn))
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1853, in apply
    return DatasetV1Adapter(super(DatasetV1, self).apply(transformation_func))
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1290, in apply
    dataset = transformation_func(self)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/experimental/ops/grouping.py", line 121, in _apply_fn
    window_size_func)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/experimental/ops/grouping.py", line 377, in __init__
    self._make_reduce_func(reduce_func, input_dataset)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/experimental/ops/grouping.py", line 424, in _make_reduce_func
    input_structure=input_structure)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2555, in __init__
    self._function = wrapper_fn._get_concrete_function_internal()
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1355, in _get_concrete_function_internal
    *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1349, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1652, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1545, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 715, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2549, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2489, in _wrapper_helper
    ret = func(*nested_args)
  File "/usr/local/lib/python3.7/site-packages/tensor2tensor-1.7.0-py3.7.egg/tensor2tensor/utils/data_reader.py", line 99, in batching_fn
    return padded_batch(grouped_dataset, batch_size, padded_shapes)
  File "/usr/local/lib/python3.7/site-packages/tensor2tensor-1.7.0-py3.7.egg/tensor2tensor/utils/data_reader.py", line 110, in padded_batch
    for name, shape in dataset.output_shapes.items()])
AttributeError: '_VariantDataset' object has no attribute 'output_shapes'

The IPA dictionary I'm using was the same I have used in the old training (that worked but was a year ago) and it is available here https://github.com/loretoparisi/docker/tree/master/g2p-seq2seq/data/dict/cmudict-ipa

homan           howˈmʌn
homann          howˈmʌn
homans          howˈmʌnz
homart          howˈmɑˌɹt
hombre          hɑˈmbɹʌ

I have used default options: g2p-seq2seq --train data/cmuipadict.txt --model_dir models/ and I'm running:

ip-192-168-22-157:g2p-seq2seq loretoparisi$ python3 -c "import tensorflow as tf; print(tf.__version__)"
1.14.0

[nshmyrev]

This is due to tesnor2tensor version incompatibility.

[loretoparisi]

@nshmyrev okay, but what I have to do to fix it? I have installed using pip3 install setupy!

[nshmyrev]

Readme says tensorflow 1.8 and tensor2tensor 1.6, then it will work.

[loretoparisi]

@nshmyrev so I have to downgrade to TF 1.8? According to the README it's at least 1.8 (so I'm getting confused here). It would be worth to specify in setup.py a specific version of TF, because by defaults it's 1.14.0 currently on pip3:

  Could not find a version that satisfies the requirement tensorflow==1.8.0 (from -r requirements.txt (line 1)) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
No matching distribution found for tensorflow==1.8.0 (from -r requirements.txt (line 1))

[loretoparisi]

@nshmyrev I have tried

setuptools>=41.0.0
tensorflow==2.0.0b1
tensor2tensor==1.6.6

but it does not work.

[ksoky]

@nshmyrev I have tried

setuptools>=41.0.0
tensorflow==2.0.0b1
tensor2tensor==1.6.6

but it does not work.

Dear Everyone, Do you have this solution yet? I have tried many versions of tensor2tensor and TensorFlow, but it still has the same problem. and for tensorflow==2.0, it has a problem without module 'tensorflow.contrib'.

[nshmyrev]

Guys, give me a week, I'll update it to much more stable pytorch.

[joazoa]

Do you have any updates on this ?

[raikarsagar]

Any updates on pytorch migration?

[nshmyrev]

Try https://github.com/hajix/G2P