error

[gitmikoy]

i ran this command: g2p-seq2seq --interactive --model_dir ./g2p-seq2seq-cmudict

then I get this error:

/usr/lib64/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "/usr/bin/g2p-seq2seq", line 9, in <module>
    load_entry_point('g2p-seq2seq==6.1.3a0', 'console_scripts', 'g2p-seq2seq')()
  File "/usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/app.py", line 107, in main
    params.hparams = g2p_trainer_utils.load_params(FLAGS.model_dir)
  File "/usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/g2p_trainer_utils.py", line 210, in load_params
    model_params = json.load(open(params_file_path))
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[mischmerz]

Same here. Tried tf from 1.0 to current. It's either missing "eager" or comes up with "No JSON object could be decoded" . Test suite fails also.

m.

[mischmerz]

I get the tests to run with tensorflow / tensor2tensor 1.5.x . However .. something seems to be wrong with model.params in g2p-seq2seq-cmudict . g2p_trainer_utils.py tries to load and de-json it, but fails. Tried to re-format it into something more json { num_layers:2, size:512 } with or without (double-) quotes results in even more cryptic errors. No joy. That's particular bad because the CMU lmtool website has problems too.

[nurtas-m]

Hello, @gitmikoy, @mischmerz The model uploaded here is invalid now. Can you, please, train new model by yourself, and then use it in interactive mode?

[gitmikoy]

@nurtas-m : I tried to train a model using the cmudict.dict file g2p-seq2seq --train cmudict.dict --model_dir hahaha/

now I get this error:

/usr/lib64/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
INFO:tensorflow:Importing user module g2p_seq2seq from path /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg
[2018-05-07 10:31:24,846] Importing user module g2p_seq2seq from path /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg
INFO:tensorflow:Overriding hparams in transformer_base with eval_drop_long_sequences=1,batch_size=4096,num_hidden_layers=3,hidden_size=256,filter_size=512,num_heads=4,length_bucket_step=1.5,max_length=30,min_length_bucket=6
[2018-05-07 10:31:24,870] Overriding hparams in transformer_base with eval_drop_long_sequences=1,batch_size=4096,num_hidden_layers=3,hidden_size=256,filter_size=512,num_heads=4,length_bucket_step=1.5,max_length=30,min_length_bucket=6
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/tensor2tensor/utils/trainer_lib.py:144: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
[2018-05-07 10:31:24,984] From /usr/lib/python2.7/site-packages/tensor2tensor/utils/trainer_lib.py:144: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
INFO:tensorflow:schedule=train_and_evaluate
[2018-05-07 10:31:24,984] schedule=train_and_evaluate
INFO:tensorflow:worker_gpu=1
[2018-05-07 10:31:24,984] worker_gpu=1
INFO:tensorflow:sync=False
[2018-05-07 10:31:24,985] sync=False
WARNING:tensorflow:Schedule=train_and_evaluate. Assuming that training is running on a single machine.
[2018-05-07 10:31:24,985] Schedule=train_and_evaluate. Assuming that training is running on a single machine.
INFO:tensorflow:datashard_devices: ['gpu:0']
[2018-05-07 10:31:24,985] datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
[2018-05-07 10:31:24,985] caching_devices: None
INFO:tensorflow:ps_devices: ['gpu:0']
[2018-05-07 10:31:24,985] ps_devices: ['gpu:0']
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x9bd7850>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'hahaha/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x9bd78d0>, '_save_summary_steps': 100}
[2018-05-07 10:31:24,985] Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x9bd7850>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'hahaha/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x9bd78d0>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x9bd5c80>) includes params argument, but params are not passed to Estimator.
[2018-05-07 10:31:24,986] Estimator's model_fn (<function wrapping_model_fn at 0x9bd5c80>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using ValidationMonitor
[2018-05-07 10:31:24,986] Using ValidationMonitor
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:279: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
[2018-05-07 10:31:24,987] From /usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:279: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/g2p_trainer_utils.py:157: __init__ (from tensorflow.contrib.learn.python.learn.experiment) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.estimator.train_and_evaluate. You will also have to convert to a tf.estimator.Estimator.
[2018-05-07 10:31:24,987] From /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/g2p_trainer_utils.py:157: __init__ (from tensorflow.contrib.learn.python.learn.experiment) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.estimator.train_and_evaluate. You will also have to convert to a tf.estimator.Estimator.
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x9bd7850>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'hahaha/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x9bd78d0>, '_save_summary_steps': 100}
[2018-05-07 10:31:24,988] Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x9bd7850>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'hahaha/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x9bd78d0>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x9c36c08>) includes params argument, but params are not passed to Estimator.
[2018-05-07 10:31:24,988] Estimator's model_fn (<function wrapping_model_fn at 0x9c36c08>) includes params argument, but params are not passed to Estimator.
Traceback (most recent call last):
  File "/usr/bin/g2p-seq2seq", line 9, in <module>
    load_entry_point('g2p-seq2seq==6.1.3a0', 'console_scripts', 'g2p-seq2seq')()
  File "/usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/app.py", line 104, in main
    g2p_model.train()
  File "/usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/g2p.py", line 242, in train
    execute_schedule(self.exp, self.params)
  File "/usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/g2p.py", line 593, in execute_schedule
    getattr(exp, params.schedule)()
  File "/usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 666, in train_and_evaluate
    self.train(delay_secs=0)
  File "/usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 389, in train
    saving_listeners=self._saving_listeners)
  File "/usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 876, in _call_train
    saving_listeners=saving_listeners)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 853, in _train_model_default
    input_fn, model_fn_lib.ModeKeys.TRAIN))
  File "/usr/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 691, in _get_features_and_labels_from_input_fn
    result = self._call_input_fn(input_fn, mode)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 798, in _call_input_fn
    return input_fn(**kwargs)
  File "/usr/lib/python2.7/site-packages/tensor2tensor/data_generators/problem.py", line 669, in estimator_input_fn
    dataset_kwargs=dataset_kwargs)
  File "/usr/lib/python2.7/site-packages/tensor2tensor/data_generators/problem.py", line 761, in input_fn
    dataset = self.dataset(**dataset_kwargs)
  File "/usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.3a0-py2.7.egg/g2p_seq2seq/g2p_problem.py", line 226, in dataset
    assert data_dir
AssertionError

I'm running on VM Centos 7 tensorflow-1.8.0: tensor2tensor-1.5.7

[nurtas-m]

@gitmikoy, Can you, please, check if there files with names "train.preprocessed" and "eval.preprocessed" created in directory hahaha/ ? Also, can you, please, clone the latest version of g2p-seq2seq (ver 6.1.4a0).

[gitmikoy]

@nurtas-m Yes those 2 files you mentioned were generated.. Here they are.. https://www.dropbox.com/s/721o7a736atpi1w/hahaha.zip?dl=0

I was able to progress.. maybe its currently training now. I ran the command like this. g2p-seq2seq --train cmudict-master/cmudict.dict --model_dir test/

Its weird because I have to put the dict file into a folder for it to work somehow. And now it runs..

/usr/lib64/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
INFO:tensorflow:Importing user module g2p_seq2seq from path /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.4a0-py2.7.egg
[2018-05-07 19:08:19,377] Importing user module g2p_seq2seq from path /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.4a0-py2.7.egg
INFO:tensorflow:Overriding hparams in transformer_base with eval_drop_long_sequences=1,batch_size=4096,num_hidden_layers=3,hidden_size=256,filter_size=512,num_heads=4,length_bucket_step=1.5,max_length=30,min_length_bucket=6
[2018-05-07 19:08:21,793] Overriding hparams in transformer_base with eval_drop_long_sequences=1,batch_size=4096,num_hidden_layers=3,hidden_size=256,filter_size=512,num_heads=4,length_bucket_step=1.5,max_length=30,min_length_bucket=6
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/tensor2tensor/utils/trainer_lib.py:144: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
[2018-05-07 19:08:21,912] From /usr/lib/python2.7/site-packages/tensor2tensor/utils/trainer_lib.py:144: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
INFO:tensorflow:schedule=train_and_evaluate
[2018-05-07 19:08:21,913] schedule=train_and_evaluate
INFO:tensorflow:worker_gpu=1
[2018-05-07 19:08:21,913] worker_gpu=1
INFO:tensorflow:sync=False
[2018-05-07 19:08:21,913] sync=False
WARNING:tensorflow:Schedule=train_and_evaluate. Assuming that training is running on a single machine.
[2018-05-07 19:08:21,913] Schedule=train_and_evaluate. Assuming that training is running on a single machine.
INFO:tensorflow:datashard_devices: ['gpu:0']
[2018-05-07 19:08:21,913] datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
[2018-05-07 19:08:21,913] caching_devices: None
INFO:tensorflow:ps_devices: ['gpu:0']
[2018-05-07 19:08:21,913] ps_devices: ['gpu:0']
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xa049950>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'test/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0xa049650>, '_save_summary_steps': 100}
[2018-05-07 19:08:21,914] Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xa049950>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'test/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0xa049650>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0xa01e050>) includes params argument, but params are not passed to Estimator.
[2018-05-07 19:08:21,914] Estimator's model_fn (<function wrapping_model_fn at 0xa01e050>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using ValidationMonitor
[2018-05-07 19:08:21,914] Using ValidationMonitor
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:279: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
[2018-05-07 19:08:21,919] From /usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:279: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.4a0-py2.7.egg/g2p_seq2seq/g2p_trainer_utils.py:157: __init__ (from tensorflow.contrib.learn.python.learn.experiment) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.estimator.train_and_evaluate. You will also have to convert to a tf.estimator.Estimator.
[2018-05-07 19:08:21,919] From /usr/lib/python2.7/site-packages/g2p_seq2seq-6.1.4a0-py2.7.egg/g2p_seq2seq/g2p_trainer_utils.py:157: __init__ (from tensorflow.contrib.learn.python.learn.experiment) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.estimator.train_and_evaluate. You will also have to convert to a tf.estimator.Estimator.
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xa049950>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'test/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0xa049650>, '_save_summary_steps': 100}
[2018-05-07 19:08:21,920] Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0xa049950>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'test/', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0xa049650>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x9c40050>) includes params argument, but params are not passed to Estimator.
[2018-05-07 19:08:21,920] Estimator's model_fn (<function wrapping_model_fn at 0x9c40050>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:Invalid symbol:name
[2018-05-07 19:08:27,209] Invalid symbol:name
WARNING:tensorflow:Invalid symbol:old
[2018-05-07 19:08:27,209] Invalid symbol:old
INFO:tensorflow:Reading data files from test/train.preprocessed
[2018-05-07 19:08:28,665] Reading data files from test/train.preprocessed
INFO:tensorflow:Calling model_fn.
[2018-05-07 19:08:29,185] Calling model_fn.
INFO:tensorflow:Setting T2TModel mode to 'train'
[2018-05-07 19:08:29,197] Setting T2TModel mode to 'train'
INFO:tensorflow:Using variable initializer: uniform_unit_scaling
[2018-05-07 19:08:29,197] Using variable initializer: uniform_unit_scaling
INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_109_256.bottom
[2018-05-07 19:08:29,250] Transforming feature 'inputs' with symbol_modality_109_256.bottom
INFO:tensorflow:Transforming 'targets' with symbol_modality_109_256.targets_bottom
[2018-05-07 19:08:29,415] Transforming 'targets' with symbol_modality_109_256.targets_bottom
INFO:tensorflow:Building model body
[2018-05-07 19:08:29,428] Building model body
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/tensor2tensor/layers/common_layers.py:600: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
[2018-05-07 19:08:29,741] From /usr/lib/python2.7/site-packages/tensor2tensor/layers/common_layers.py:600: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
INFO:tensorflow:Transforming body output with symbol_modality_109_256.top
[2018-05-07 19:08:33,768] Transforming body output with symbol_modality_109_256.top
WARNING:tensorflow:From /usr/lib/python2.7/site-packages/tensor2tensor/layers/common_layers.py:1828: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

[2018-05-07 19:08:33,873] From /usr/lib/python2.7/site-packages/tensor2tensor/layers/common_layers.py:1828: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

INFO:tensorflow:Base learning rate: 0.200000
[2018-05-07 19:08:33,947] Base learning rate: 0.200000
INFO:tensorflow:Trainable Variables Total size: 3981568
[2018-05-07 19:08:33,975] Trainable Variables Total size: 3981568
INFO:tensorflow:Using optimizer Adam
[2018-05-07 19:08:33,975] Using optimizer Adam
/usr/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Done calling model_fn.
[2018-05-07 19:08:40,538] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
[2018-05-07 19:08:40,539] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
[2018-05-07 19:08:43,523] Graph was finalized.
2018-05-07 19:08:43.609704: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
INFO:tensorflow:Running local_init_op.
[2018-05-07 19:08:44,902] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
[2018-05-07 19:08:44,993] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 1 into test/model.ckpt.
[2018-05-07 19:09:19,747] Saving checkpoints for 1 into test/model.ckpt.
INFO:tensorflow:loss = 5.7692256, step = 1
[2018-05-07 19:09:20,929] loss = 5.7692256, step = 1
INFO:tensorflow:global_step/sec: 0.107042
[2018-05-07 19:24:55,341] global_step/sec: 0.107042
INFO:tensorflow:loss = 3.6330156, step = 101 (934.633 sec)
[2018-05-07 19:24:55,563] loss = 3.6330156, step = 101 (934.633 sec)
INFO:tensorflow:global_step/sec: 0.134358
[2018-05-07 19:37:19,521] global_step/sec: 0.134358
INFO:tensorflow:loss = 3.0366695, step = 201 (744.000 sec)
[2018-05-07 19:37:19,562] loss = 3.0366695, step = 201 (744.000 sec)
INFO:tensorflow:global_step/sec: 0.137393
[2018-05-07 19:49:27,345] global_step/sec: 0.137393
INFO:tensorflow:loss = 2.7166274, step = 301 (727.807 sec)
[2018-05-07 19:49:27,369] loss = 2.7166274, step = 301 (727.807 sec)
INFO:tensorflow:global_step/sec: 0.136914
[2018-05-07 20:01:37,641] global_step/sec: 0.136914
INFO:tensorflow:loss = 3.0419216, step = 401 (730.280 sec)
[2018-05-07 20:01:37,649] loss = 3.0419216, step = 401 (730.280 sec)
INFO:tensorflow:global_step/sec: 0.137773
[2018-05-07 20:13:43,474] global_step/sec: 0.137773
INFO:tensorflow:loss = 2.5091295, step = 501 (725.838 sec)
[2018-05-07 20:13:43,488] loss = 2.5091295, step = 501 (725.838 sec)
INFO:tensorflow:global_step/sec: 0.138012
[2018-05-07 20:25:48,070] global_step/sec: 0.138012
INFO:tensorflow:loss = 1.9631003, step = 601 (724.584 sec)
[2018-05-07 20:25:48,071] loss = 1.9631003, step = 601 (724.584 sec)
INFO:tensorflow:global_step/sec: 0.135116
[2018-05-07 20:38:08,306] global_step/sec: 0.135116
INFO:tensorflow:loss = 1.6986619, step = 701 (740.309 sec)
[2018-05-07 20:38:08,380] loss = 1.6986619, step = 701 (740.309 sec)
INFO:tensorflow:global_step/sec: 0.130907
[2018-05-07 20:50:52,054] global_step/sec: 0.130907
INFO:tensorflow:loss = 1.2037007, step = 801 (763.687 sec)
[2018-05-07 20:50:52,068] loss = 1.2037007, step = 801 (763.687 sec)
INFO:tensorflow:global_step/sec: 0.130416
[2018-05-07 21:03:38,831] global_step/sec: 0.130416
INFO:tensorflow:loss = 1.2126111, step = 901 (766.766 sec)
[2018-05-07 21:03:38,833] loss = 1.2126111, step = 901 (766.766 sec)
INFO:tensorflow:global_step/sec: 0.135104
[2018-05-07 21:15:59,001] global_step/sec: 0.135104
INFO:tensorflow:loss = 0.9130957, step = 1001 (740.170 sec)
[2018-05-07 21:15:59,003] loss = 0.9130957, step = 1001 (740.170 sec)

Does it take long time to train a model?

[nurtas-m]

@gitmikoy,

I was able to progress.. maybe its currently training now. I ran the command like this. g2p-seq2seq --train cmudict-master/cmudict.dict --model_dir test/

Its weird because I have to put the dict file into a folder for it to work somehow.

Are you sure, that in first time when you run the program there was cmudict.dict file in your current directory? When you launch the program, you have to specify the path to the dictionary, but not just a name of the dictionary: g2p-seq2seq --train path/to/the/dictionary --model_dir hahaha

Does it take long time to train a model?

Depends on how accurate model do you want to receive. In my experiences you can get the best model after ~70.000 steps (on 1 GPU it takes about 2 hours). The training will stop after 200.000 steps. You may, manually stop the training process at any moment.

[nurtas-m]

@gitmikoy, Is the problem solved now?

[gitmikoy]

@nurtas-m Sorry I was not able to reply right away.. I was sure the file cmudict.dict was there when in the terminal because I use the tab key to find it.. but when the cmudict is inside the folder things seem to work like the command I used above.

I'm currently training the model on cpu only (takes very long time) and targeting to 70,000 steps like you suggested. Thank you nurtas for helping me :)

[nshmyrev]

Duplicate of issue #115