search

carpedm20 / multi-speaker-tacotron-tensorflow

Multi-speaker Tacotron in TensorFlow.
http://carpedm20.github.io/tacotron
Other
631 stars 342 forks source link

학습 스텝 완료 후 에러(Could not synthesize characters to Hangul.) #78

Open froglike6 opened 4 years ago

froglike6 commented 4 years ago

Exiting due to exception: Could not synthesize characters to Hangul. Traceback (most recent call last): File "train.py", line 268, in train log_dir, step, loss, "train") File "train.py", line 104, in save_and_plot parallel_run(fn, items, parallel=False) File "/root/tts/multi-speaker-tacotron-tensorflow/utils/init.py", line 151, in parallel_run out = fn(item) File "train.py", line 95, in save_and_plot_fn skip_eos_and_pad=True, combine_jamo=True)) File "/root/tts/multi-speaker-tacotron-tensorflow/text/init.py", line 77, in sequence_to_text return jamo_to_korean(result) File "/root/tts/multi-speaker-tacotron-tensorflow/text/korean.py", line 64, in jamo_to_korean new_text += _get_text_from_candidates(candidates) File "/root/tts/multi-speaker-tacotron-tensorflow/text/korean.py", line 53, in _get_text_from_candidates return j2h(*dict(zip(["lead", "vowel", "tail"], candidates))) File "/usr/local/lib/python3.6/site-packages/jamo/jamo.py", line 290, in j2h return jamo_to_hangul(lead, vowel, tail) File "/usr/local/lib/python3.6/site-packages/jamo/jamo.py", line 277, in jamo_to_hangul '\x00') jamo.jamo.InvalidJamoError: Could not synthesize characters to Hangul. 2020-05-18 10:29:45.504211: W tensorflow/core/kernels/queue_base.cc:295] _0_datafeeder/input_queue_1: Skipping cancelled enqueue attempt with queue not closed 2020-05-18 10:29:45.504287: W tensorflow/core/kernels/queue_base.cc:295] _1_datafeeder/input_queue: Skipping cancelled enqueue attempt with queue not closed Traceback (most recent call last): Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(args) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(*args) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn status, run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn status, run_metadata) File "/usr/lib64/python3.6/contextlib.py", line 88, in exit next(self.gen) File "/usr/lib64/python3.6/contextlib.py", line 88, in exit next(self.gen) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/loss_coeff_0_3, _arg_datafeeder/mel_targets_0_4, _arg_datafeeder/linear_targets_0_2)]] tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[Node: datafeeder/input_queue_1_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue_1, _arg_datafeeder/inputs_1_0_1, _arg_datafeeder/input_lengths_1_0_0, _arg_datafeeder/loss_coeff_1_0_3, _arg_datafeeder/mel_targets_1_0_4, _arg_datafeeder/linear_targets_1_0_2)]]

During handling of the above exception, another exception occurred:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): Traceback (most recent call last): File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 204, in run self._enqueue_next_group() File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 204, in run self._enqueue_next_group() File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 242, in _enqueue_next_group self._session.run(self._enqueue_op, feed_dict=feed_dict) File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 242, in _enqueue_next_group self._session.run(self._enqueue_op, feed_dict=feed_dict) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run options, run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run options, run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/loss_coeff_0_3, _arg_datafeeder/mel_targets_0_4, _arg_datafeeder/linear_targets_0_2)]]

Caused by op 'datafeeder/input_queue_enqueue', defined at: File "train.py", line 322, in main() File "train.py", line 318, in main train(config.model_dir, config) File "train.py", line 136, in train data_type='train', batch_size=hparams.batch_size) File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 160, in init self._enqueue_op = queue.enqueue(self._placeholders) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 321, in enqueue self._queue_ref, vals, name=scope) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1587, in _queue_enqueue_v2 name=name) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/loss_coeff_0_3, _arg_datafeeder/mel_targets_0_4, _arg_datafeeder/linear_targets_0_2)]]

tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[Node: datafeeder/input_queue_1_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue_1, _arg_datafeeder/inputs_1_0_1, _arg_datafeeder/input_lengths_1_0_0, _arg_datafeeder/loss_coeff_1_0_3, _arg_datafeeder/mel_targets_1_0_4, _arg_datafeeder/linear_targets_1_0_2)]]

Caused by op 'datafeeder/input_queue_1_enqueue', defined at: File "train.py", line 322, in main() File "train.py", line 318, in main train(config.model_dir, config) File "train.py", line 139, in train data_type='test', batch_size=config.num_test) File "/root/tts/multi-speaker-tacotron-tensorflow/datasets/datafeeder.py", line 160, in init self._enqueue_op = queue.enqueue(self._placeholders) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 321, in enqueue self._queue_ref, vals, name=scope) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1587, in _queue_enqueue_v2 name=name) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled [[Node: datafeeder/input_queue_1_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](datafeeder/input_queue_1, _arg_datafeeder/inputs_1_0_1, _arg_datafeeder/input_lengths_1_0_0, _arg_datafeeder/loss_coeff_1_0_3, _arg_datafeeder/mel_targets_1_0_4, _arg_datafeeder/linear_targets_1_0_2)]]

위와 같이 뜹니다.. 어떻게 해야 하나요?

solalala-12 commented 4 years ago

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데

froglike6 commented 4 years ago

학습 후 파일 저장 시 이 에러가 뜹니다

2020년 5월 18일 (월) 오후 5:51, LEE SORA notifications@github.com님이 작성:

synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carpedm20/multi-speaker-tacotron-tensorflow/issues/78#issuecomment-630040827, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

solalala-12 commented 4 years ago

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA notifications@github.com님이 작성: … synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

https://github.com/carpedm20/multi-speaker-tacotron-tensorflow/issues/10 참고해보세요~

froglike6 commented 4 years ago

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA notifications@github.com님이 작성: … synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

10 참고해보세요~

이 문제 해결했더니 list index out of range 에러뜨네요..

solalala-12 commented 4 years ago

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA notifications@github.com님이 작성: … synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

10 참고해보세요~

이 문제 해결했더니 list index out of range 에러뜨네요..

train.py 를 실행했더니 Could not synthesize characters to Hangul. 에러가 뜨신건가요 ? list index out of range 에러도 상세히 올려주시면 답변 드리겠습니다.

froglike6 commented 4 years ago

학습 후 파일 저장 시 이 에러가 뜹니다 2020년 5월 18일 (월) 오후 5:51, LEE SORA notifications@github.com님이 작성: … synthesize 어떤 문구 하셨나요? input으로 들어온 텍스트를 변환하는 과정에서 생기는 문제같은데 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHOQVVNV7KHGXPUWGFLRSDZKFANCNFSM4NDU5AKA .

10 참고해보세요~

이 문제 해결했더니 list index out of range 에러뜨네요..

train.py 를 실행했더니 Could not synthesize characters to Hangul. 에러가 뜨신건가요 ? list index out of range 에러도 상세히 올려주시면 답변 드리겠습니다.

[root@dhmowiki tts]# python3 train.py --data_path=datasets/ksw [] MODEL dir: logs/ksw_2020-05-20_13-36-08 [] PARAM path: logs/ksw_2020-05-20_13-36-08/params.json

[!] Detect non-krbook dataset. Set sampling rate from 24000 to 20000

[*] git recv-parse HEAD: da58cf1ea59211e036a1d9af09f8dc7c10f191a9

================================================== [*] dit diff: diff --git a/datasets/generate_data.py b/datasets/generate_data.py index b351eb2..7afbb1d 100644 --- a/datasets/generate_data.py +++ b/datasets/generate_data.py @@ -37,7 +37,7 @@ def build_from_path(config):

 loss_coeff = defaultdict(one)
 if config.metadata_path.endswith("json"):

================================================== [] Checkpoint path: logs/ksw_2020-05-20_13-36-08/model.ckpt [] Loading training data from: ['datasets/ksw/data'] [*] Using model: logs/ksw_2020-05-20_13-36-08 Hyperparameters: adam_beta1: 0.9 adam_beta2: 0.999 attention_size: 256 attention_state_size: 256 attention_type: bah_mon batch_size: 16 cleaners: korean_cleaners dec_layer_num: 2 dec_prenet_sizes: [256, 128] dec_rnn_size: 256 decay_learning_rate_mode: 0 dropout_prob: 0.8 embedding_size: 256 enc_bank_channel_size: 128 enc_bank_size: 16 enc_highway_depth: 4 enc_maxpool_width: 2 enc_prenet_sizes: [256, 128] enc_proj_sizes: [128, 128] enc_proj_width: 3 enc_rnn_size: 128 frame_length_ms: 50 frame_shift_ms: 12.5 griffin_lim_iters: 60 ignore_recognition_level: 1 initial_data_greedy: True initial_learning_rate: 0.002 initial_phase_step: 8000 main_data: [''] main_data_greedy_factor: 0 max_iters: 200 min_iters: 30 min_level_db: -100 min_tokens: 50 model_type: single num_freq: 1025 num_mels: 80 post_bank_channel_size: 256 post_bank_size: 8 post_highway_depth: 4 post_maxpool_width: 2 post_proj_sizes: [256, 80] post_proj_width: 3 post_rnn_size: 256 power: 1.5 preemphasis: 0.97 prioritize_loss: False recognition_loss_coeff: 0.2 reduction_factor: 4 ref_level_db: 20 sample_rate: 24000 skip_inadequate: False speaker_embedding_size: 16 use_fixed_test_inputs: False filter_by_min_max_frame_batch: 100%|███████████| 41/41 [00:00<00:00, 924.70it/s] [datasets/ksw/data] Loaded metadata for 11 examples (0.01 hours) [datasets/ksw/data] Max length: 737 [datasets/ksw/data] Min length: 229

{'datasets/ksw/data': 1.0}

filter_by_min_max_frame_batch: 100%|███████████| 41/41 [00:00<00:00, 800.97it/s] [datasets/ksw/data] Loaded metadata for 11 examples (0.01 hours) [datasets/ksw/data] Max length: 737 [datasets/ksw/data] Min length: 229

{'datasets/ksw/data': 1.0}

======================================== model_type: single

Initialized Tacotron model. Dimensions: embedding: 256 speaker embedding: None prenet out: 128 encoder out: 256 attention out: 256 concat attn & out: 512 decoder cell out: 256 decoder out (4 frames): 320 decoder out (1 frame): 80 postnet out: 512 linear out: 1025

model_type: single

Initialized Tacotron model. Dimensions: embedding: 256 speaker embedding: None prenet out: 128 encoder out: 256 attention out: 256 concat attn & out: 512 decoder cell out: 256 decoder out (4 frames): 320 decoder out (1 frame): 80 postnet out: 512 linear out: 1025 2020-05-20 13:36:15.699815: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2020-05-20 13:36:15.699851: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2020-05-20 13:36:15.699860: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2020-05-20 13:36:15.699867: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2020-05-20 13:36:15.699875: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. Starting new training run at commit: None Generated 2 batches of size 2 in 0.000 sec Traceback (most recent call last): File "/root/tts/tts/datasets/datafeeder.py", line 204, in run self._enqueue_next_group() File "/root/tts/tts/datasets/datafeeder.py", line 229, in _enqueue_nextgroup for in range(int(n self._batches_per_group // len(self.datadirs)))] File "/root/tts/tts/datasets/datafeeder.py", line 229, in for in range(int(n self._batches_per_group // len(self.data_dirs)))] File "/root/tts/tts/datasets/datafeeder.py", line 257, in _get_next_example data_path = data_paths[self._offset[data_dir]] IndexError: list index out of range 원래 트레인 실행하면 hangul 에러 떴는데 hangul 에러 해결하고 다시 실행하니 out of range 에러가 뜹니다.

solalala-12 commented 4 years ago

Loaded metadata for 11 examples을 보니 사용할 데이터는 11개 밖에 없는 것 같네요. 다음 batch를 가져오지 못해서 나오는 에러인 것 같습니다. 경로에 데이터가 잘 있는지 확인해보시거나 batch_size를 조절하거나 batches_per_group을 조절해보세요.

froglike6 commented 4 years ago

어떻게 조절하는지 아시나요? batch size는 train.py에서 조절하는 것 같은데 32와 8 두가지가 있네요...

2020년 5월 20일 (수) 오후 1:49, LEE SORA notifications@github.com님이 작성:

Loaded metadata for 11 examples을 보니 사용할 데이터는 11개 밖에 없는 것 같네요. 다음 batch를 가져오지 못해서 나오는 에러인 것 같습니다. 경로에 데이터가 잘 있는지 확인해보시거나 batch_size를 조절하거나 batches_per_group을 조절해보세요.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carpedm20/multi-speaker-tacotron-tensorflow/issues/78#issuecomment-631234308, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO25MHLHA2O6CXRINHH7NLLRSNOOFANCNFSM4NDU5AKA .

solalala-12 commented 4 years ago

batch_size는 hparams.py에서 수정하시면 됩니다.