search

p0p4k / vits2_pytorch

unofficial vits2-TTS implementation in pytorch
https://arxiv.org/abs/2307.16430
MIT License
465 stars 81 forks source link

KSS dataset for 490 epochs but the quality is not as good as I expected #49

Closed ggpid closed 10 months ago

ggpid commented 10 months ago

First of all, thank you for sharing such a wonderful code. I trained using the KSS dataset for 490 epochs, but the quality is not as good as I expected. It seems that the TTS speaks a bit fast. wav What might have gone wrong during the training?

{ "train": { "log_interval": 200, "eval_interval": 3000, "seed": 1234, "epochs": 20000, "learning_rate": 2e-4, "betas": [0.8, 0.99], "eps": 1e-9, "batch_size": 32, "fp16_run": false, "lr_decay": 0.999875, "segment_size": 8192, "init_lr_ratio": 1, "warmup_epochs": 0, "c_mel": 45, "c_kl": 1.0, "fft_sizes": [384, 683, 171], "hop_sizes": [30, 60, 10], "win_lengths": [150, 300, 60], "window": "hann_window" }, "data": { "use_mel_posterior_encoder": true, "training_files":"kss/kss_cjke_train.txt.cleaned", "validation_files":"kss/kss_cjke_val.txt.cleaned", "text_cleaners":["cjke_cleaners2"], "max_wav_value": 32768.0, "sampling_rate": 44100, "filter_length": 1024, "hop_length": 256, "win_length": 1024, "n_mel_channels": 80, "mel_fmin": 0.0, "mel_fmax": null, "add_blank": true, "n_speakers": 0, "cleaned_text": true }, "model": { "use_mel_posterior_encoder": true, "use_transformer_flows": true, "transformer_flow_type": "pre_conv", "use_spk_conditioned_encoder": false, "use_noise_scaled_mas": true, "use_duration_discriminator": true, "ms_istft_vits": false, "mb_istft_vits": true, "istft_vits": false, "subbands": 4, "gen_istft_n_fft": 16, "gen_istft_hop_size": 4, "inter_channels": 192, "hidden_channels": 96, "filter_channels": 768, "n_heads": 2, "n_layers": 3, "kernel_size": 3, "p_dropout": 0.1, "resblock": "1", "resblock_kernel_sizes": [3,7,11], "resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]], "upsample_rates": [4,4], "upsample_initial_channel": 256, "upsample_kernel_sizes": [16,16], "n_layers_q": 3, "use_spectral_norm": false, "use_sdp": false } }

p0p4k commented 10 months ago

try add blank false, use_duration_discriminator false. also, the following configs are not from this repo. 

"ms_istft_vits": false,
"mb_istft_vits": true,
"istft_vits": false,
"subbands": 4,
"gen_istft_n_fft": 16,
"gen_istft_hop_size": 4,

If you are using this repo, it will automatically ignore those flags anyway.

p0p4k commented 10 months ago

Another way to debug the issue in this model -> 1) load the model 2) send a mel_spec to model.enc_q and get stats, sample 3) send the sampled output to model.dec and generate wav|

If the wav is bad quality or fast, the problem might be there. If the wav is ok, then our issue is in enc_p and duration_predictor.

ggpid commented 10 months ago

@p0p4k I posted in the wrong repo, but I got important information. Thank you.

p0p4k commented 10 months ago

Really nice comment by @w11wo https://github.com/FENRlR/MB-iSTFT-VITS2/issues/4#issuecomment-1718954124