Vocoder can't adapt to TTS Model

[Wangzhen-kris]

Hello, I tried to train taco2 and vocoder separately and I can't synthesize effective voices with the pre-trained vocoder. I'm very strange that use griffin lim, the synthesized voice seems to be pretty good. Here synthesized voice with WaveGrad and GL: https://drive.google.com/file/d/1fAf7SGdnWfODQsCgVYBFd2l1g0ZWy1fA/view?usp=sharing https://drive.google.com/file/d/1AVXXRXarlZQ2pmYmg-pI-oUNu9Nx2IHP/view?usp=sharing

Here config.json of Taco2: ` { "model": "Tacotron2", "run_name": "ayq", "run_description": "tacotron2 with DDC and differential spectral loss.",

// AUDIO PARAMETERS
"audio":{
    // stft parameters
    "fft_size": 1024,         // number of stft frequency levels. Size of the linear spectogram frame.
    "win_length": 1024,      // stft window length in ms.
    "hop_length": 256,       // stft window hop-lengh in ms.
    "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
    "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

    // Audio processing parameters
    "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate.
    "preemphasis": 0.0,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
    "ref_level_db": 20,     // reference level db, theoretically 20db is the sound of air.

    // Silence trimming
    "do_trim_silence": false,// enable trimming of slience of audio as you load it. LJspeech (true), TWEB (false), Nancy (true)
    "trim_db": 60,          // threshold for timming silence. Set this according to your dataset.

    // Griffin-Lim
    "power": 1.5,           // value to sharpen wav signals after GL algorithm.
    "griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.

    // MelSpectrogram parameters
    "num_mels": 80,         // size of the mel spec frame.
    "mel_fmin": 50.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
    "mel_fmax": 7600.0,     // maximum freq level for mel-spec. Tune for dataset!!
    "spec_gain": 1,

    // Normalization parameters
    "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
    "min_level_db": -100,   // lower bound for normalization
    "symmetric_norm": true, // move normalization to range [-1, 1]
    "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
    "clip_norm": true,      // clip normalized values into the range.
    "stats_path": null    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},

// VOCABULARY PARAMETERS
// if custom character set is not defined,
// default set in symbols.py is used
// "characters":{
//     "pad": "_",
//     "eos": "~",
//     "bos": "^",
//     "characters": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!'(),-.:;? ",
//     "punctuations":"!'(),-.:;? ",
//     "phonemes":"iyɨʉɯuɪʏʊeøɘəɵɤoɛœɜɞʌɔæɐaɶɑɒᵻʘɓǀɗǃʄǂɠǁʛpbtdʈɖcɟkɡqɢʔɴŋɲɳnɱmʙrʀⱱɾɽɸβfvθðszʃʒʂʐçʝxɣχʁħʕhɦɬɮʋɹɻjɰlɭʎʟˈˌːˑʍwɥʜʢʡɕʑɺɧɚ˞ɫ"
// },

// DISTRIBUTED TRAINING
"distributed":{
    "backend": "nccl",
    "url": "tcp:\/\/localhost:54321"
},

"reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

// TRAINING
"batch_size": 32,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
"eval_batch_size":16,
"r": 7,                 // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.
"gradual_training": null,   //[[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 32], [290000, 1, 32]], //set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled. For Tacotron, you might need to reduce the 'batch_size' as you proceeed.
"mixed_precision": false,     // level of optimization with NVIDIA's apex feature for automatic mixed FP16/FP32 precision (AMP), NOTE: currently only O1 is supported, and use "O1" to activate.

// LOSS SETTINGS
"loss_masking": true,       // enable / disable loss masking against the sequence padding.
"decoder_loss_alpha": 0.5,  // original decoder loss weight. If > 0, it is enabled
"postnet_loss_alpha": 0.25, // original postnet loss weight. If > 0, it is enabled
"postnet_diff_spec_alpha": 0.25,     // differential spectral loss weight. If > 0, it is enabled
"decoder_diff_spec_alpha": 0.25,     // differential spectral loss weight. If > 0, it is enabled
"decoder_ssim_alpha": 0.5,     // decoder ssim loss weight. If > 0, it is enabled
"postnet_ssim_alpha": 0.25,     // postnet ssim loss weight. If > 0, it is enabled
"ga_alpha": 5.0,           // weight for guided attention loss. If > 0, guided attention is enabled.
"stopnet_pos_weight": 15.0, // pos class weight for stopnet loss since there are way more negative samples than positive samples.

// VALIDATION
"run_eval": true,
"test_delay_epochs": 10,  //Until attention is aligned, testing only wastes computation time.
"test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

// OPTIMIZER
"noam_schedule": false,        // use noam warmup and lr schedule.
"grad_clip": 1.0,              // upper limit for gradients for clipping.
"epochs": 100000,                // total number of epochs to train.
"lr": 0.0001,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
"wd": 0.000001,                // Weight decay weight.
"warmup_steps": 4000,          // Noam decay steps to increase the learning rate from 0 to "lr"
"seq_len_norm": false,         // Normalize eash sample loss with its length to alleviate imbalanced datasets. Use it if your dataset is small or has skewed distribution of sequence lengths.

// TACOTRON PRENET
"memory_size": -1,             // ONLY TACOTRON - size of the memory queue used fro storing last decoder predictions for auto-regression. If < 0, memory queue is disabled and decoder only uses the last prediction frame.
"prenet_type": "original",     // "original" or "bn".
"prenet_dropout": false,       // enable/disable dropout at prenet.

// TACOTRON ATTENTION
"attention_type": "original",  // 'original' or 'graves'
"attention_heads": 4,          // number of attention heads (only for 'graves')
"attention_norm": "sigmoid",   // softmax or sigmoid.
"windowing": false,            // Enables attention windowing. Used only in eval mode.
"use_forward_attn": false,     // if it uses forward attention. In general, it aligns faster.
"forward_attn_mask": false,    // Additional masking forcing monotonicity only in eval mode.
"transition_agent": false,     // enable/disable transition agent of forward attention.
"location_attn": true,         // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
"bidirectional_decoder": false,  // use https://arxiv.org/abs/1907.09006. Use it, if attention does not work well with your dataset.
"double_decoder_consistency": true,  // use DDC explained here https://erogol.com/solving-attention-problems-of-tts-models-with-double-decoder-consistency-draft/
"ddc_r": 7,                           // reduction rate for coarse decoder.

// STOPNET
"stopnet": true,               // Train stopnet predicting the end of synthesis.
"separate_stopnet": true,      // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.

// TENSORBOARD and LOGGING
"print_step": 25,       // Number of steps to log training on console.
"tb_plot_step": 100,    // Number of steps to plot TB training figures.
"print_eval": false,     // If True, it prints intermediate loss values in evalulation.
"save_step": 10000,      // Number of training steps expected to save traninpg stats and checkpoints.
"checkpoint": true,     // If true, it saves checkpoints per "save_step"
"tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

// DATA LOADING
"text_cleaner": "basic_cleaners",
"enable_eos_bos_chars": false, // enable/disable beginning of sentence and end of sentence chars.
"num_loader_workers": 4,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
"num_val_loader_workers": 4,    // number of evaluation data loader processes.
"batch_group_size": 4,  //Number of batches to shuffle after bucketing.
"min_seq_len": 6,       // DATASET-RELATED: minimum text length to use in training
"max_seq_len": 220,     // DATASET-RELATED: maximum text length

// PATHS
"output_path": "/data1/mozillawz/result_aqy",

// PHONEMES
"phoneme_cache_path": "/data1/mozillawz/mozilla_pinyin/Phoneme_cache/",  // phoneme computation is slow, therefore, it caches results in the given folder.
"use_phonemes": false,           // use phonemes instead of raw characters. It is suggested for better pronounciation.
"phoneme_language": "en-us",     // depending on your target language, pick one from  https://github.com/bootphon/phonemizer#languages

// MULTI-SPEAKER and GST
"use_speaker_embedding": true,      // use speaker embedding to enable multi-speaker learning.
"use_gst": false,                       // use global style tokens
"use_external_speaker_embedding_file": false, // if true, forces the model to use external embedding per sample instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
"external_speaker_embedding_file": "../../speakers-vctk-en.json", // if not null and use_external_speaker_embedding_file is true, it is used to load a specific embedding file and thus uses these embeddings instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
"gst":  {                           // gst parameter if gst is enabled
    "gst_style_input": null,        // Condition the style input either on a
                                    // -> wave file [path to wave] or
                                    // -> dictionary using the style tokens {'token1': 'value', 'token2': 'value'} example {"0": 0.15, "1": 0.15, "5": -0.15}
                                    // with the dictionary being len(dict) <= len(gst_style_tokens).
    "gst_embedding_dim": 512,
    "gst_num_heads": 4,
    "gst_style_tokens": 10,
    "gst_use_speaker_embedding": false
},

// DATASETS
"datasets":   // List of datasets. They all merged and they get different speaker_ids.
    [
        {
            "name": "own",
            "path": "/data1/mozillawz/",
            "meta_file_train": "train_aqy.txt", // for vtck if list, ignore speakers id in list for train, its useful for test cloning with new speakers
            "meta_file_val": "val_aqy.txt"
        }
    ]

} `

Here vocoder_config.json of WaveGrad: ` { "run_name": "wavegrad-aqy", "run_description": "wavegrad aqy",

"audio":{
    "fft_size": 1024,         // number of stft frequency levels. Size of the linear spectogram frame.
    "win_length": 1024,      // stft window length in ms.
    "hop_length": 256,       // stft window hop-lengh in ms.
    "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
    "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

    // Audio processing parameters
    "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
    "preemphasis": 0.0,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
    "ref_level_db": 0,     // reference level db, theoretically 20db is the sound of air.

    // Silence trimming
    "do_trim_silence": false,// enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
    "trim_db": 60,          // threshold for timming silence. Set this according to your dataset.

    // MelSpectrogram parameters
    "num_mels": 80,         // size of the mel spec frame.
    "mel_fmin": 50.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
    "mel_fmax": 7600.0,     // maximum freq level for mel-spec. Tune for dataset!!
    "spec_gain": 1.0,         // scaler value appplied after log transform of spectrogram.

    // Normalization parameters
    "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
    "min_level_db": -100,   // lower bound for normalization
    "symmetric_norm": true, // move normalization to range [-1, 1]
    "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
    "clip_norm": true,      // clip normalized values into the range.
    "stats_path": ""    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},

// DISTRIBUTED TRAINING
"mixed_precision": true,     // enable torch mixed precision training (true, false)
"distributed":{
    "backend": "nccl",
    "url": "tcp:\/\/localhost:54322"
},

"target_loss": "avg_wavegrad_loss",  // loss value to pick the best model to save after each epoch

// MODEL PARAMETERS
"generator_model": "wavegrad",
"model_params":{
    "use_weight_norm": true,
    "y_conv_channels":32,
    "x_conv_channels":768,
    "ublock_out_channels": [512, 512, 256, 128, 128],
    "dblock_out_channels": [128, 128, 256, 512],
    "upsample_factors": [4, 4, 4, 2, 2],
    "upsample_dilations": [
        [1, 2, 1, 2],
        [1, 2, 1, 2],
        [1, 2, 4, 8],
        [1, 2, 4, 8],
        [1, 2, 4, 8]]
},

// DATASET
"data_path": "/data1/wangzhen/data_aishell/AISHELL_and_Originbeat/",  // root data path. It finds all wav files recursively from there.
"feature_path": null,   // if you use precomputed features
"seq_len": 6144,        // 24 * hop_length
"pad_short": 0,      // additional padding for short wavs
"conv_pad": 0,          // additional padding against convolutions applied to spectrograms
"use_noise_augment": false,     // add noise to the audio signal for augmentation
"use_cache": false,      // use in memory cache to keep the computed features. This might cause OOM.

"reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

// TRAINING
"batch_size": 64,      // Batch size for training.

// NOISE SCHEDULE PARAMS - Only effective at training time.
"train_noise_schedule":{
    "min_val": 1e-6,
    "max_val": 1e-2,
    "num_steps": 1000
},
"test_noise_schedule":{
    "min_val": 1e-6,
    "max_val": 1e-2,
    "num_steps": 50
},

// VALIDATION
"run_eval": true,       // enable/disable evaluation run

// OPTIMIZER
"epochs": 1000000,                // total number of epochs to train.
"clip_grad": 1.0,                 // Generator gradient clipping threshold. Apply gradient clipping if > 0
"lr_scheduler": "MultiStepLR",  // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"lr_scheduler_params": {
    "gamma": 0.5,
    "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
},
"lr": 1e-4,                  // Initial learning rate. If Noam decay is active, maximum learning rate.

// TENSORBOARD and LOGGING
"print_step": 50,       // Number of steps to log traning on console.
"print_eval": false,     // If True, it prints loss values for each step in eval run.
"save_step": 5000,      // Number of training steps expected to plot training stats on TB and save model checkpoints.
"checkpoint": true,     // If true, it saves checkpoints per "save_step"
"tb_model_param_stats": true,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

// DATA LOADING
"num_loader_workers": 4,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
"num_val_loader_workers": 4,    // number of evaluation data loader processes.
"eval_split_size": 256,

// PATHS
"output_path": "/data1/mozillawz/wavegrad_models/"

} `

[Wangzhen-kris]

I found the ref_level_db in audio is a key parameter by extra experiments, I should set the same value in the two config files. At present, the new training process is in progress, and I will continue to update the latest results here.

[Wangzhen-kris]

I found the ref_level_db in audio is a key parameter by extra experiments, I should set the same value in the two config files. At present, the new training process is in progress, and I will continue to update the latest results here.

Disappointingly, I retrained wavegrad and it still doesn't work until now.

[lexkoro]

could you share screens of your tensorboard logging?

[Wangzhen-kris]

could you share screens of your tensorboard logging?

@SanjaESC Sure, I will share the log files of tacotron2 and wavegrad, I also share screens to simplify matters for you.

Here the log file of Tacotron2: https://drive.google.com/file/d/1Ex6voGEqEcDFt5Wt7wSR1H_gEyF4edXu/view?usp=sharing

Here the log file of WaveGrad: https://drive.google.com/file/d/12jwQy6ngKtrbA3pdwKLWCmFfmpqrtU_a/view?usp=sharing

Here the screens of Tacotron2:

Here the screens of WaveGrad:

[lexkoro]

Looking at the Train and Eval figures in the TTS training the energy level of the frequency seems weird (to high?). Maybe because it was trained with spec_gain: 1, but not quite sure here.

Other than that, the voice samples sound fine. How do you synthesize with WaveGrad? Maybe that's where the problem lies.

[Wangzhen-kris]

Looking at the Train and Eval figures in the TTS training the energy level of the frequency seems weird (to high?). Maybe because it was trained with spec_gain: 1, but not quite sure here.

Other than that, the voice samples sound fine. How do you synthesize with WaveGrad? Maybe that's where the problem lies.

Thank you for your prompt reply. Next I will focus on spec_gain, and it's weird that wavegrad can also synthesize effective voice from gt mel. Maybe I should train another vocoder with the same configuration in future, and then, I will continue to update the latest results here.

[albluc24]

Have you tryed retraining both models with scale_stats.npy computed by TTS.bin.compute_statistics? They say also not to use it with multi speaker models, but none of my vocoders worked without the shared stats file and I got the same results as you.

[Wangzhen-kris]

Have you tryed retraining both models with scale_stats.npy computed by TTS.bin.compute_statistics? They say also not to use it with multi speaker models, but none of my vocoders worked without the shared stats file and I got the same results as you.

@albluc24 As you said, I trained multi speaker model, so I didn't use scale_stats.npy. Have you not solved this problem yet？

[albluc24]

I did train my multi speaker model with scalestats, and taht's the only way I got it working

[lexkoro]

Next I will focus on spec_gain

for me 20 worked quite well.

Regarding wavegrad, how do you synthesize the speech? Maybe there is an error and your model is okay.

[Wangzhen-kris]

Next I will focus on spec_gain

for me 20 worked quite well.

Regarding wavegrad, how do you synthesize the speech? Maybe there is an error and your model is okay.

You are right, I got it wrong. I reconfirmed and found there is an error in wavegrad here. I referred to #518 , although it can run without error, the synthesized speech is invalid. Consequently, I'll try to train another vocoder with reasonable spec_gain.

[Wangzhen-kris]

I trained parallel-wavegan and it‘s able to adapt to the TTS model, so the reason for this issue is wavegrad.

[Wangzhen-kris]

@SanjaESC Hi, I tried to fix the bug of wavegrad to synthesize voice, but it didn't work. Is there anyone else making this effort? And I can't find an active issue with this error.

[lexkoro]

What error exactly do you mean? Are you executing it correctly? What are the steps? It works fine for me.

[Wangzhen-kris]

What error exactly do you mean? Are you executing it correctly? What are the steps? It works fine for me.

The error is that I can't synthesize effective voice with wavegrad. After getting your reply, I compared my files with the latest master branch and found the difference in synthesize.py. Here is mine, but I can't remember why I did it.

    if not use_gl:
        # Use if not computed noise schedule with tune_wavegrad
        beta = np.linspace(1e-6, 0.01, 50)
        vocoder_model.compute_noise_level(beta)
        device_type = "cuda" if use_cuda else "cpu"

        # Here is the reason for the error!!! 
        vocoder_input = ap._normalize(mel_postnet_spec.T)

        waveform = vocoder_model.inference(torch.FloatTensor(vocoder_input).to(device_type).unsqueeze(0))

After the fix, it worked.

[lexkoro]

So it works now?

[Wangzhen-kris]

So it works now?

Yes! Thank you @SanjaESC , i'd close the issue soon.