Error while finetuning multi speaker from pretrained model

[ken2190]

I'm trying to finetune from pretrained model pflow-2000.ckpt on custom multi speaker dataset in German, in the first 16 epoch we trained without issue, but now i get many RuntimeError like below. Does anyone has same issue?

Total estimated model params size (MB): 128
Restored all states from the checkpoint at /home/ubuntu/DATA/pflowtts_pytorch/logs/train/multispeaker/runs/2024-02-24_20-42-51/checkpoints/last.ckpt
Exception ignored in: <function Image.__del__ at 0x7f3623636560>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 4056, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
...
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f3623636560>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 4056, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f3623636560>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 4056, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f36237d2dd0>
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/tkinter/__init__.py", line 388, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Epoch 17/-2 ━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━ 376/799 1:23:05 • 1:21:07 0.09it/s v_num: 0.000 loss/train_step: 16.598

[p0p4k]

What about training from scratch? Does it give the same error?

[ken2190]

it so strange, it now disappeared, Could you add instruction for inferencing trained model (single and multi speaker)

i run this but it doesnt work

python pflow/cli.py \
--file "/home/ubuntu/DATA/Custom_Voice_Training/deep_clone_voice/tools/radtts/processed_speech.txt" \
--checkpoint_path "/home/ubuntu/DATA/pflowtts_pytorch/logs/train/multispeaker/runs/2024-02-27_08-30-57/checkpoints/last.ckpt" \
--vocoder "hifigan_univ_v1" --spk 60 

[!] Configurations:
        - Model: pflow_ljspeech
        - Vocoder: hifigan_univ_v1
        - Temperature: 0.667
        - Speaking rate: 1.0
        - Number of ODE steps: 10
        - Speaker: 60
Traceback (most recent call last):
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflow/cli.py", line 418, in <module>
    cli()
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflow/cli.py", line 273, in cli
    paths = assert_required_models_available(args)
  File "/home/ubuntu/DATA/pflowtts_pytorch/pflow/cli.py", line 77, in assert_required_models_available
    assert_model_downloaded(model_path, PFLOW_URLS[args.model])
KeyError: 'pflow_ljspeech'

with vctk model

(/home/ubuntu/DATA/pflowtts_pytorch/pflowtts_env) ubuntu@t4:~/DATA/pflowtts_pytorch$ python pflow/cli.py --model "pflow_vctk" --file "/home/ubuntu/DATA/Custom_Voice_Training/deep_clone_voice/tools/radtts/processed_speech.txt" --checkpoint_path "/home/ubuntu/DATA/pflowtts_pytorch/logs/train/multispeaker/runs/2024-02-27_08-30-57/checkpoints/last.ckpt" --vocoder "hifigan_univ_v1" --spk 60
usage: cli.py [-h] [--model {}] [--checkpoint_path CHECKPOINT_PATH] [--vocoder {hifigan_T2_v1,hifigan_univ_v1}] [--text TEXT]
              [--file FILE] [--spk SPK] [--temperature TEMPERATURE] [--speaking_rate SPEAKING_RATE] [--steps STEPS] [--cpu]
              [--denoiser_strength DENOISER_STRENGTH] [--output_folder OUTPUT_FOLDER] [--batched] [--batch_size BATCH_SIZE]
cli.py: error: argument --model: invalid choice: 'pflow_vctk' (choose from )

[p0p4k]

The model does not take in speaker id as input, cause it is going to inference from a sample prompt of the speaker. Try using the notebook for synthesis first and then modify this cli code accordingly. I will fix it later on when I get some time. Good luck!

[isjwdu]

I have the same error... I was training multi-speaker models from scratch by VCTK dataset in epoch 0. Everything works well on my server, but when I moved this project to my personal computer(WSL2 Ubuntu20.04, 13600kf + 4090), this error happened. I was still trying to figure out a solution.