local variable 'eval_steps_per_epoch' referenced before assignment

[MostafaAlaviyan]

hi I want to train fastspeech2 on a new dataset, I extract duration by MFA. when I start the training, after 499 epochs when the model wants to do the evaluation step the bellow error was accrued:

[train]:   0%|▎                                                                                                                                                    | 492/200000 [02:33<13:39:21,  4.06it/s]2021-12-27 08:48:37,582 (base_trainer:140) INFO: (Steps: 492) Finished 492 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 493/200000 [02:33<13:46:12,  4.02it/s]2021-12-27 08:48:37,835 (base_trainer:140) INFO: (Steps: 493) Finished 493 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 494/200000 [02:34<13:26:53,  4.12it/s]2021-12-27 08:48:38,064 (base_trainer:140) INFO: (Steps: 494) Finished 494 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 495/200000 [02:34<14:44:48,  3.76it/s]2021-12-27 08:48:38,385 (base_trainer:140) INFO: (Steps: 495) Finished 495 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 496/200000 [02:34<14:02:28,  3.95it/s]2021-12-27 08:48:38,608 (base_trainer:140) INFO: (Steps: 496) Finished 496 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 497/200000 [02:34<13:37:46,  4.07it/s]2021-12-27 08:48:38,838 (base_trainer:140) INFO: (Steps: 497) Finished 497 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 498/200000 [02:35<13:36:29,  4.07it/s]2021-12-27 08:48:39,082 (base_trainer:140) INFO: (Steps: 498) Finished 498 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 499/200000 [02:35<14:59:40,  3.70it/s]2021-12-27 08:48:39,411 (base_trainer:140) INFO: (Steps: 499) Finished 499 epoch training (1 steps per epoch).
[train]:   0%|▎                                                                                                                                                    | 500/200000 [02:35<14:17:50,  3.88it/s]2021-12-27 08:48:39,638 (base_trainer:883) INFO: (Steps: 500) Start evaluation.
                          2021-12-27 08:48:39.732004: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1345] No whitelist ops found, nothing to do
2021-12-27 08:48:39.738364: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1345] No whitelist ops found, nothing to do
[eval]: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "examples/fastspeech2/train_fastspeech2.py", line 417, in <module>
    main()
  File "examples/fastspeech2/train_fastspeech2.py", line 409, in main
    resume=args.resume,
  File "/root/anaconda3/envs/tts2/lib/python3.6/site-packages/tensorflow_tts/trainers/base_trainer.py", line 999, in fit
    self.run()
  File "/root/anaconda3/envs/tts2/lib/python3.6/site-packages/tensorflow_tts/trainers/base_trainer.py", line 103, in run
    self._train_epoch()
  File "/root/anaconda3/envs/tts2/lib/python3.6/site-packages/tensorflow_tts/trainers/base_trainer.py", line 129, in _train_epoch
    self._check_eval_interval()
  File "/root/anaconda3/envs/tts2/lib/python3.6/site-packages/tensorflow_tts/trainers/base_trainer.py", line 166, in _check_eval_interval
    self._eval_epoch()
  File "/root/anaconda3/envs/tts2/lib/python3.6/site-packages/tensorflow_tts/trainers/base_trainer.py", line 897, in _eval_epoch
    f"(Steps: {self.steps}) Finished evaluation "
UnboundLocalError: local variable 'eval_steps_per_epoch' referenced before assignment
[train]:   0%|▎                                                                                                                                                    | 500/200000 [02:36<17:21:13,  3.19it/s]

how can I fix the error?

[MostafaAlaviyan]

@dathudeptrai

[MostafaAlaviyan]

@ZDisket

[dathudeptrai]

@MostafaAlaviyan you need debug by yourself why this bug happened (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/tensorflow_tts/trainers/base_trainer.py#L897), i don't see anything wrong :D

[MostafaAlaviyan]

ok, thanks, I can fixed the error.

[sphiNur]

@MostafaAlaviyan

ok, thanks, I can fixed the error.

---

I have same problem

2022-05-16 08:45:26,618 (base_trainer:888) INFO: (Steps: 500) Start evaluation.
[eval]: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "examples/tacotron2/train_tacotron2.py", line 528, in <module>
    main()
  File "examples/tacotron2/train_tacotron2.py", line 516, in main
    trainer.fit(
  File "/home/nurzat/miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_tts/trainers/base_trainer.py", line 1010, in fit
    self.run()
  File "/home/nurzat/miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_tts/trainers/base_trainer.py", line 104, in run
    self._train_epoch()
  File "/home/nurzat/miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_tts/trainers/base_trainer.py", line 130, in _train_epoch
    self._check_eval_interval()
  File "/home/nurzat/miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_tts/trainers/base_trainer.py", line 167, in _check_eval_interval
    self._eval_epoch()
  File "/home/nurzat/miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_tts/trainers/base_trainer.py", line 902, in _eval_epoch
    f"(Steps: {self.steps}) Finished evaluation "
UnboundLocalError: local variable 'eval_steps_per_epoch' referenced before assignment

can you tell me how to fix it?

[MostafaAlaviyan]

@Qarnur . please see the ''' https://github.com/TensorSpeech/TensorFlowTTS/issues/480#issuecomment-768942886 '''