Synthesis on same reference wav has different pitch during test and inference times

[george-roussos]

It might not be a bug, but I have noticed the same reference .wav yields different results during test and inference times. When training and testing the same sentence using the same wav file, the speech is much more natural, but when using the same text and same wav file in synthesis.py, the pitch is worse. I looked around the code but could not find anything.

Branch dev-gst-embeddings

[erogol]

I don't understand the problem. Is it synthesis.py? How do you run the inference which works good?

[george-roussos]

Yeah, so during training, when the epoch ends, the same sentence is synthesized (after evaluation) using the same reference wav file that I use when I run synthesize.py outside training. And these two produce different results. If I scale down by 0.4 in compute_gst in tacotron_abstact.py, the results are closer to each other, but they are still different.

[lexkoro]

Can you share your config.json? Checking this line in train_tts.py it tries to get they key "_style_wav_fortest" which was replaced with "_gst_styleinput" in the config. So maybe you are not using a ref wav during training at all.

Also it's maybe better to use the dev branch since it has all the changes from dev-gst-embeddings + fixes

[george-roussos]

// AUDIO PARAMETERS
"audio":{
    // stft parameters
    "fft_size": 1100,         // number of stft frequency levels. Size of the linear spectogram frame.
    "win_length": 1100,      // stft window length in ms.
    "hop_length": 275,       // stft window hop-lengh in ms.
    "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
    "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

    // Audio processing parameters
    "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate.
    "preemphasis": 0.98,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
    "ref_level_db": 20,     // reference level db, theoretically 20db is the sound of air.

    // Silence trimming
    "do_trim_silence": false,   // enable trimming of slience of audio as you load it. LJspeech (true), TWEB (false), Nancy (true)
    "trim_db": 60,          // threshold for timming silence. Set this according to your dataset.

    // Griffin-Lim
    "power": 1.5,           // value to sharpen wav signals after GL algorithm.
    "griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.

    // MelSpectrogram parameters
    "num_mels": 80,         // size of the mel spec frame.
    "mel_fmin": 0.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
    "mel_fmax": 8000.0,     // maximum freq level for mel-spec. Tune for dataset!!
    "spec_gain": 20.0,

    // Normalization parameters
    "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
    "min_level_db": -100,   // lower bound for normalization
    "symmetric_norm": true, // move normalization to range [-1, 1]
    "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
    "clip_norm": true,      // clip normalized values into the range.
    "stats_path": null    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},

// VOCABULARY PARAMETERS
// if custom character set is not defined,
// default set in symbols.py is used
// "characters":{
//     "pad": "_",
//     "eos": "~",
//     "bos": "^",
//     "characters": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!'(),-.:;? ",
//     "punctuations":"!'(),-.:;? ",
//     "phonemes":"iyɨʉɯuɪʏʊeøɘəɵɤoɛœɜɞʌɔæɐaɶɑɒᵻʘɓǀɗǃʄǂɠǁʛpbtdʈɖcɟkɡqɢʔɴŋɲɳnɱmʙrʀⱱɾɽɸβfvθðszʃʒʂʐçʝxɣχʁħʕhɦɬɮʋɹɻjɰlɭʎʟˈˌːˑʍwɥʜʢʡɕʑɺɧɚ˞ɫ"
// },

// DISTRIBUTED TRAINING
"distributed":{
    "backend": "nccl",
    "url": "tcp:\/\/localhost:54321"
},

"reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

// TRAINING
"batch_size": 32,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
"eval_batch_size":40,
"r": 7,                 // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.
"gradual_training": [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 32]], //set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled. For Tacotron, you might need to reduce the 'batch_size' as you proceeed.
"loss_masking": true,         // enable / disable loss masking against the sequence padding.
"ga_alpha": 10.0,        // weight for guided attention loss. If > 0, guided attention is enabled.
"apex_amp_level": null, // level of optimization with NVIDIA's apex feature for automatic mixed FP16/FP32 precision (AMP), NOTE: currently only O1 is supported, and use "O1" to activate.

// VALIDATION
"run_eval": true,
"test_delay_epochs": 3,  //Until attention is aligned, testing only wastes computation time.
"test_sentences_file": "/home/georgios.roussos/tests.txt",  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

// OPTIMIZER
"noam_schedule": false,        // use noam warmup and lr schedule.
"grad_clip": 1.0,              // upper limit for gradients for clipping.
"epochs": 1000,                // total number of epochs to train.
"lr": 0.0001,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
"wd": 0.000001,                // Weight decay weight.
"warmup_steps": 4000,          // Noam decay steps to increase the learning rate from 0 to "lr"
"seq_len_norm": true,         // Normalize eash sample loss with its length to alleviate imbalanced datasets. Use it if your dataset is small or has skewed distribution of sequence lengths.

// TACOTRON PRENET
"memory_size": -1,             // ONLY TACOTRON - size of the memory queue used fro storing last decoder predictions for auto-regression. If < 0, memory queue is disabled and decoder only uses the last prediction frame.
"prenet_type": "original",           // "original" or "bn".
"prenet_dropout": true,       // enable/disable dropout at prenet.

// TACOTRON ATTENTION
"attention_type": "original",  // 'original' or 'graves'
"attention_heads": 4,          // number of attention heads (only for 'graves')
"attention_norm": "softmax",   // softmax or sigmoid.
"windowing": false,            // Enables attention windowing. Used only in eval mode.
"use_forward_attn": true,     // if it uses forward attention. In general, it aligns faster.
"forward_attn_mask": true,    // Additional masking forcing monotonicity only in eval mode.
"transition_agent": true,     // enable/disable transition agent of forward attention.
"location_attn": true,         // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
"bidirectional_decoder": false,  // use https://arxiv.org/abs/1907.09006. Use it, if attention does not work well with your dataset.
"double_decoder_consistency": false,  // use DDC explained here https://erogol.com/solving-attention-problems-of-tts-models-with-double-decoder-consistency-draft/
"ddc_r": 7,                           // reduction rate for coarse decoder.

// STOPNET
"stopnet": true,               // Train stopnet predicting the end of synthesis.
"separate_stopnet": true,      // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.

// TENSORBOARD and LOGGING
"print_step": 50,       // Number of steps to log training on console.
"tb_plot_step": 100,    // Number of steps to plot TB training figures.
"print_eval": false,     // If True, it prints intermediate loss values in evalulation.
"save_step": 10000,      // Number of training steps expected to save traninpg stats and checkpoints.
"checkpoint": true,     // If true, it saves checkpoints per "save_step"
"tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

// DATA LOADING
"text_cleaner": "swedish_cleaners",
"enable_eos_bos_chars": false, // enable/disable beginning of sentence and end of sentence chars.
"num_loader_workers": 8,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
"num_val_loader_workers": 8,    // number of evaluation data loader processes.
"batch_group_size": 0,  //Number of batches to shuffle after bucketing.
"min_seq_len": 1,       // DATASET-RELATED: minimum text length to use in training
"max_seq_len": 400,     // DATASET-RELATED: maximum text length

// PATHS
"output_path": "/home/georgios.roussos/",

// PHONEMES
"phoneme_cache_path": "/home/georgios.roussos/TTS/karin_phones",  // phoneme computation is slow, therefore, it caches results in the given folder.
"use_phonemes": true,           // use phonemes instead of raw characters. It is suggested for better pronounciation.
"phoneme_language": "sv",     // depending on your target language, pick one from  https://github.com/bootphon/phonemizer#languages

// MULTI-SPEAKER and GST
"use_speaker_embedding": false,      // use speaker embedding to enable multi-speaker learning.
"use_external_speaker_embedding_file": false, // if true, forces the model to use external embedding per sample instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
"external_speaker_embedding_file": "../../speakers-vctk-en.json", // if not null and use_external_speaker_embedding_file is true, it is used to load a specific embedding file and thus uses these embeddings instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
"use_gst": true,                    // use global style tokens
"gst":  {                           // gst parameter if gst is enabled
    "gst_style_input": "/Users/george.roussos/Desktop/gst_ref/59.wav",        // Condition the style input either on a 
                                    // -> wave file [path to wave] or 
                                    // -> dictionary using the style tokens {'token1': 'value', 'token2': 'value'} example {"0": 0.15, "1": 0.15, "5": -0.15}  
                                    // with the dictionary being len(dict) <= len(gst_style_tokens).
    "gst_embedding_dim": 256,       
    "gst_num_heads": 4,
    "gst_style_tokens": 10
},  

// DATASETS
"datasets":   // List of datasets. They all merged and they get different speaker_ids.
    [
        {
            "name": "ljspeech",
            "path": "/home/georgios.roussos/karin_shortened_sils",
            "meta_file_train": "metadata.csv", // for vtck if list, ignore speakers id in list for train, its useful for test cloning with new speakers
            "meta_file_val": "val.csv"
        }
    ]}

[george-roussos]

Can you share your config.json? Checking this line in train_tts.py it tries to get they key "_style_wav_fortest" which was replaced with "_gst_styleinput" in the config. So maybe you are not using a ref wav during training at all.

Also it's maybe better to use the dev branch since it has all the changes from dev-gst-embeddings + fixes

Aha, that is interesting. I saw the value was changed. Does it mean it takes zeros as input? I couldn't figure it out, because the same setup yielded different results on the same dataset, as regards the GST learning. So the same wav file on the same sentence, gave different pitch and now it looks like any .wav file I provide synthesize.py with, the pitch is too high.

[lexkoro]

Yes, if no style input is passed it uses zeros. So without passing a ref. wav the pitch is fine?

[george-roussos]

Yes, if no style input is passed it uses zeros. So without passing a ref. wav the pitch is fine?

It is! It is neutral but not too neutral. Providing a wav file (any wav file), makes the pitch too high. If I scale by 0.4, it is better. Conditioning on certain tokens is also fine, but the attention is a bit worse.

[erogol]

maybe @Edresson can take a look?

[erogol]

Yes, if no style input is passed it uses zeros. So without passing a ref. wav the pitch is fine?

It is! It is neutral but not too neutral. Providing a wav file (any wav file), makes the pitch too high. If I scale by 0.4, it is better. Conditioning on certain tokens is also fine, but the attention is a bit worse.

why 0.4 ? how did you come up with that?

[george-roussos]

when the epoch ends,

means in evaluate() function where the test sentences are synthesized?

Yes! This one.

why 0.4 ? how did you come up with that?

I have been reading this paper and since they talked about scaling by multiplying and I think it is also how conditioning on specific tokens now happens on the dev branch, I thought I would try this. So I tried many values, from 0.05 to 0.9, but they were either too low pitched and there was a lot of glottalization, or the pitch was too high and the attention also broke. 0.4 gave me a nice middle ground. 0.4 produces natural speech with natural prosody and attention is retained almost wholly when synthesizing a text that is 2261 characters (approximately two pages off a book). But I think it depends on the voice. Before, I was training on @Edresson's voice-cloning branch of his TTS fork and my results were also great. Which is why I thought it was weird that the exact same config, the exact same text, with the same reference wav now gives a different prosody. Any wav file I am trying produces speech that is very high pitched, like my speaker is shouting.

[Edresson]

@george-roussos I think that when you synthesized during training, as c.get("style_wav_for_test") returns NONE the GST received a zero tensor as input. Hence the difference in training time and during synthesis. Try again now and let us know if the same wav style generates the same result.

I checked the code and the only thing I found different between synthesis and training in GST is the conversion to float32 of the spectrogram and the wav. The dataloader does this and the compute_style_mel function did not, but I believe that this does not change much. But I sent PR #499 anyway.

Are you training the multispeaker GST? Currently the GST with multispeaker is very unstable, as the GST learns characteristics of the speaker and this causes problems.

[george-roussos]

Nope, I am training single. I did a pull and tried to synthesize the speech again using synthesis.py, but the result is still the same (the speech sounds less natural without the scaling by 0.4, compared to the voice-cloning branch). I have always used the same dataset, with the same hparams and the same sentence. Has something changed in the way the GST layer learns prosody?

[Edresson]

Can you supply two notebook colabs with the synthesis of the two models for debugging? it would help me find the problem. As far as I know the two implementations are exactly the same, gst_layers is the same embedding is concatenated in the same place, everything looks the same.

[george-roussos]

I made two notebooks, with synthesis.py of voice-cloning here and dev-gst-embeddings here. I can make notebooks with the synthesize.py instead if you need it, but I am not allowed to upload my model :(

[Edresson]

Maybe you can use this notebook to change the checkpoint and see if the model synthesizes correctly. In it the synthesis function is imported directly, so maybe we can try to find out where the problem is!

The checkpoint on that notebook is compatible with my voice-cloning branch, so you can change the branch and do the tests using the same checkpoint (an open checkpoint) so we can find the problem! Sorry for not doing the tests myself, but I have little time available!

[george-roussos]

No problem! I tried to run inference on the voice-cloning branch, but it wouldn't work, first it kept complaining about some entries in the config.json file, but when I fixed these, I got errors about missing some keys:

Traceback (most recent call last): File "synthesize.py", line 159, in <module> model.load_state_dict(cp['model']) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1045, in load_state_dict self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for Tacotron2: Missing key(s) in state_dict: "encoder.convolutions.0.net.0.weight", "encoder.convolutions.0.net.0.bias", "encoder.convolutions.0.net.1.weight", "encoder.convolutions.0.net.1.bias", "encoder.convolutions.0.net.1.running_mean", "encoder.convolutions.0.net.1.running_var", "encoder.convolutions.1.net.0.weight", "encoder.convolutions.1.net.0.bias", "encoder.convolutions.1.net.1.weight", "encoder.convolutions.1.net.1.bias", "encoder.convolutions.1.net.1.running_mean", "encoder.convolutions.1.net.1.running_var", "encoder.convolutions.2.net.0.weight", "encoder.convolutions.2.net.0.bias", "encoder.convolutions.2.net.1.weight", "encoder.convolutions.2.net.1.bias", "encoder.convolutions.2.net.1.running_mean", "encoder.convolutions.2.net.1.running_var", "decoder.prenet.layers.0.linear_layer.weight", "decoder.prenet.layers.1.linear_layer.weight", "decoder.attention.location_layer.location_conv.weight", "postnet.convolutions.0.net.0.weight", "postnet.convolutions.0.net.0.bias", "postnet.convolutions.0.net.1.weight", "postnet.convolutions.0.net.1.bias", "postnet.convolutions.0.net.1.running_mean", "postnet.convolutions.0.net.1.running_var", "postnet.convolutions.1.net.0.weight", "postnet.convolutions.1.net.0.bias", "postnet.convolutions.1.net.1.weight", "postnet.convolutions.1.net.1.bias", "postnet.convolutions.1.net.1.running_mean", "postnet.convolutions.1.net.1.running_var", "postnet.convolutions.2.net.0.weight", "postnet.convolutions.2.net.0.bias", "postnet.convolutions.2.net.1.weight", "postnet.convolutions.2.net.1.bias", "postnet.convolutions.2.net.1.running_mean", "postnet.convolutions.2.net.1.running_var", "postnet.convolutions.3.net.0.weight", "postnet.convolutions.3.net.0.bias", "postnet.convolutions.3.net.1.weight", "postnet.convolutions.3.net.1.bias", "postnet.convolutions.3.net.1.running_mean", "postnet.convolutions.3.net.1.running_var", "postnet.convolutions.4.net.0.weight", "postnet.convolutions.4.net.0.bias", "postnet.convolutions.4.net.1.weight", "postnet.convolutions.4.net.1.bias", "postnet.convolutions.4.net.1.running_mean", "postnet.convolutions.4.net.1.running_var". Unexpected key(s) in state_dict: "encoder.convolutions.0.convolution1d.weight", "encoder.convolutions.0.convolution1d.bias", "encoder.convolutions.0.batch_normalization.weight", "encoder.convolutions.0.batch_normalization.bias", "encoder.convolutions.0.batch_normalization.running_mean", "encoder.convolutions.0.batch_normalization.running_var", "encoder.convolutions.0.batch_normalization.num_batches_tracked", "encoder.convolutions.1.convolution1d.weight", "encoder.convolutions.1.convolution1d.bias", "encoder.convolutions.1.batch_normalization.weight", "encoder.convolutions.1.batch_normalization.bias", "encoder.convolutions.1.batch_normalization.running_mean", "encoder.convolutions.1.batch_normalization.running_var", "encoder.convolutions.1.batch_normalization.num_batches_tracked", "encoder.convolutions.2.convolution1d.weight", "encoder.convolutions.2.convolution1d.bias", "encoder.convolutions.2.batch_normalization.weight", "encoder.convolutions.2.batch_normalization.bias", "encoder.convolutions.2.batch_normalization.running_mean", "encoder.convolutions.2.batch_normalization.running_var", "encoder.convolutions.2.batch_normalization.num_batches_tracked", "decoder.prenet.linear_layers.0.linear_layer.weight", "decoder.prenet.linear_layers.1.linear_layer.weight", "decoder.attention.location_layer.location_conv1d.weight", "postnet.convolutions.0.convolution1d.weight", "postnet.convolutions.0.convolution1d.bias", "postnet.convolutions.0.batch_normalization.weight", "postnet.convolutions.0.batch_normalization.bias", "postnet.convolutions.0.batch_normalization.running_mean", "postnet.convolutions.0.batch_normalization.running_var", "postnet.convolutions.0.batch_normalization.num_batches_tracked", "postnet.convolutions.1.convolution1d.weight", "postnet.convolutions.1.convolution1d.bias", "postnet.convolutions.1.batch_normalization.weight", "postnet.convolutions.1.batch_normalization.bias", "postnet.convolutions.1.batch_normalization.running_mean", "postnet.convolutions.1.batch_normalization.running_var", "postnet.convolutions.1.batch_normalization.num_batches_tracked", "postnet.convolutions.2.convolution1d.weight", "postnet.convolutions.2.convolution1d.bias", "postnet.convolutions.2.batch_normalization.weight", "postnet.convolutions.2.batch_normalization.bias", "postnet.convolutions.2.batch_normalization.running_mean", "postnet.convolutions.2.batch_normalization.running_var", "postnet.convolutions.2.batch_normalization.num_batches_tracked", "postnet.convolutions.3.convolution1d.weight", "postnet.convolutions.3.convolution1d.bias", "postnet.convolutions.3.batch_normalization.weight", "postnet.convolutions.3.batch_normalization.bias", "postnet.convolutions.3.batch_normalization.running_mean", "postnet.convolutions.3.batch_normalization.running_var", "postnet.convolutions.3.batch_normalization.num_batches_tracked", "postnet.convolutions.4.convolution1d.weight", "postnet.convolutions.4.convolution1d.bias", "postnet.convolutions.4.batch_normalization.weight", "postnet.convolutions.4.batch_normalization.bias", "postnet.convolutions.4.batch_normalization.running_mean", "postnet.convolutions.4.batch_normalization.running_var", "postnet.convolutions.4.batch_normalization.num_batches_tracked". size mismatch for decoder.attention_rnn.weight_ih: copying a param with shape torch.Size([4096, 1024]) from checkpoint, the shape in current model is torch.Size([4096, 896]). size mismatch for decoder.attention.inputs_layer.linear_layer.weight: copying a param with shape torch.Size([128, 768]) from checkpoint, the shape in current model is torch.Size([128, 640]). size mismatch for decoder.decoder_rnn.weight_ih: copying a param with shape torch.Size([4096, 1792]) from checkpoint, the shape in current model is torch.Size([4096, 1664]). size mismatch for decoder.linear_projection.linear_layer.weight: copying a param with shape torch.Size([560, 1792]) from checkpoint, the shape in current model is torch.Size([560, 1664]).

[Edresson]

@george-roussos I have 1 checkpoint compatible with both branchs and 2 configs. I actually checked the branch dev-gst-embeddings and my "voice-cloning" branch synthesizes different audios for the same GST sample, and on my branch the audio sounds much better. I believe we have a bug in the GST inference on that branch (on the dev branch too). I am looking for the problem again, as soon as I find and solve it I make a new PR and notice here.

Thanks for reporting the problem :).

[Edresson]

@george-roussos I have 1 checkpoint compatible with both branchs and 2 configs. I actually checked the branch dev-gst-embeddings and my "voice-cloning" branch synthesizes different audios for the same GST sample, and on my branch the audio sounds much better. I believe we have a bug in the GST inference on that branch (on the dev branch too). I am looking for the problem again, as soon as I find and solve it I make a new PR and notice here.

Thanks for reporting the problem :).

This difference in my tests was caused by the conversion from db to amp and from amp to db. In the current version this has been changed, so checkpoints trained with gst before the update are incompatible. So if you try to load an old model, just updating config.json it will not work. This only applies to models trained with GST as the other models do not need to extract spectrograms in the synthesis time

@george-roussos I didn't find anything that could cause what you reported other than the change made by @SanjaESC . Without a replicable example it is very difficult to find the problem ..

[george-roussos]

Sorry for the double notification. Anyway, it just sounds really amplified when I give it the wav. Like it scales the style Mel it computes *2. If I scale by 0.4 as I wrote, the prosody is very natural and closely resembles that of the model trained on the voice-cloning branch. Because I am training a new model on dev-gst-embeddings, so there shouldn’t be compatibility issues (i.e. I’m not trying to use my checkpoint from the other branch). And my hparams have all stayed the same. Also, it’s a problem with every wav I give it.

I will look again myself but I think it’s weird that it only happens with wavs (specific tokens works ok) and I use the exact same data. It’s a shame because a wav makes the speech very lively, but it still is okay because 0.4 works great. Thanks for looking anyway 😃

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts